Star Flow: Accelerating On-Line Processing of Astronomic Data on Heterogeneous Processors

Several new astronomy projects, including the Large Synoptic Survey Telescope (LSST) in the US, and the Ground-based Wide-Angle Cameras (GWAC) in China, are building either a single, large telescope or an array of smaller cameras to continuously observe a considerable portion of the sky. Once in operation, these instruments will produce observation images every few seconds. Furthermore, complex processing is required to catalog raw images online, and transient objects, such as stars that change positions or brightness, should be detected from the data instantly to enable timely follow-up observation. Consequently, accelerating the online processing of these data is crucial to meet the demands of these projects and to facilitate astronomic discoveries.

There have been initial studies on utilizing commodity computer processors such as multicore CPUs (Central Processing Units) and GPUs (Graphics Processing Units) to parallelize computation on astronomical data. However, the pipeline processing of camera data during normal observations is complex, involving image processing, astrometry computation, and database operations. As a result, parallelizing this pipeline remains a challenging open problem, especially in a heterogeneous environment with both multicore CPUs and GPUs. Moreover, recent developments in heterogeneous processors, such as the Intel Phi co-processors for the CPU, and the dynamic scheduling capabilities in NVIDIA Kepler GPUs to involve multiple CPU cores, pose interesting opportunities to parallel computing.

We propose Star Flow, an open-source software project that utilizes state-of-the-art many-core CPUs with their co-processors as well as GPUs to parallelize the processing pipeline of astronomical observation data. In Star Flow, we will tackle three major components in the processing pipeline: (1) source extraction, (2) image subtraction, and (3) cross match. In source extraction, the tabular attributes of hundreds of thousands of stars and galaxies are extracted from each observation image. Image subtraction and cross match are two alternatives of finding transient objects with the difference being whether the input are observation images or tabular data.

Documents

Xiaoying Jia, Qiong Luo: Multi-Assignment Single Joins for Parallel Cross-Match of Astronomic Catalogs on Heterogeneous Clusters. SSDBM 2016: 12:1-12:12
Xiaoying Jia, Qiong Luo, Dongwei Fan: Cross-Matching Large Astronomical Catalogs on Heterogeneous Clusters. ICPADS 2015: 617-624
Accelerating astronomical source extraction with graphics processors. Baoxue Zhao, MPhil Thesis, HKUST, Dec 2014.
Parallelizing Astronomical Source Extraction on the GPU. Baoxue Zhao, Qiong Luo, and Chao Wu. IEEE 9th International Conference on e-Science, Beijing, China, Oct 2013, pp 88-97.
Accelerating Astronomical Image Subtraction on Heterogeneous Processors. Yan Zhao, Qiong Luo, Senhong Wang, and Chao Wu. IEEE 9th International Conference on e-Science, Beijing, China, Oct 2013, pp 70-77.
Accelerating In-Memory Cross Match of Astronomical Catalogs. Senhong Wang, Yan Zhao, Qiong Luo, Chao Wu, and Yang Xv. IEEE 9th International Conference on e-Science, Beijing, China, Oct 2013, pp 326-333.

Software

GPU-SExtractor. The source code package of our GPU-based version of the original SExtractor. Jan 2015. Jan 2015.
P-HOTPANTS. The source code package of our GPU and multicore-CPU parallelized version of the original HOTPANTS image subtraction tool by Andrew Becker. May 2014.

Software License

The license is a free non-exclusive, non-transferable license to reproduce, use, modify and display the source code version of the Software, with or without modifications solely for non-commercial research, educational or evaluation purposes. The license does not entitle Licensee to technical support, telephone assistance, enhancements or updates to the Software. All rights, title to and ownership interest in Software, including all intellectual property rights therein shall remain in HKUST.

Acknowledgement

We thank our collaborator the National Astronomical Observatories of China for providing us application requirements, access to their data sets, and sharing their domain knowledge. Funding for this project is provided by grants 616012, 617509, and 16206414 from the Hong Kong Research Grants Council and MRA11EG01 from Microsoft SQL Server China R&D.