online read us now
Paper details
Number 3 - September 2019
Volume 29 - 2019
Efficient astronomical data condensation using approximate nearest neighbors
Szymon Łukasik, Konrad Lalik, Piotr Sarna, Piotr A. Kowalski, Małgorzata Charytanowicz, Piotr Kulczycki
Abstract
Extracting useful information from astronomical observations represents one of the most challenging tasks of data exploration. This is largely due to the volume of the data acquired using advanced observational tools. While other challenges
typical for the class of big data problems (like data variety) are also present, the size of datasets represents the most significant obstacle in visualization and subsequent analysis. This paper studies an efficient data condensation algorithm aimed at providing its compact representation. It is based on fast nearest neighbor calculation using tree structures and parallel processing. In addition to that, the possibility of using approximate identification of neighbors, to even further improve the algorithm time performance, is also evaluated. The properties of the proposed approach, both in terms of performance and condensation quality, are experimentally assessed on astronomical datasets related to the GAIA mission. It is concluded that the introduced technique might serve as a scalable method of alleviating the problem of the dataset size.
Keywords
big data, astronomy, data reduction, nearest neighbor search, kd-trees