Star Flow: Accelerating On-Line Processing of Astronomic Data on Heterogeneous Processors

Several new astronomy projects, including the Large Synoptic Survey Telescope (LSST) in the US, and the Ground-based Wide-Angle Cameras (GWAC) in China, are building either a single, large telescope or an array of smaller cameras to continuously observe a considerable portion of the sky. Once in operation, these instruments will produce observation images every few seconds. Furthermore, complex processing is required to catalog raw images online, and transient objects, such as stars that change positions or brightness, should be detected from the data instantly to enable timely follow-up observation. Consequently, accelerating the online processing of these data is crucial to meet the demands of these projects and to facilitate astronomic discoveries.

There have been initial studies on utilizing commodity computer processors such as multicore CPUs (Central Processing Units) and GPUs (Graphics Processing Units) to parallelize computation on astronomical data. However, the pipeline processing of camera data during normal observations is complex, involving image processing, astrometry computation, and database operations. As a result, parallelizing this pipeline remains a challenging open problem, especially in a heterogeneous environment with both multicore CPUs and GPUs. Moreover, recent developments in heterogeneous processors, such as the Intel Phi co-processors for the CPU, and the dynamic scheduling capabilities in NVIDIA Kepler GPUs to involve multiple CPU cores, pose interesting opportunities to parallel computing.

We propose Star Flow, an open-source software project that utilizes state-of-the-art many-core CPUs with their co-processors as well as GPUs to parallelize the processing pipeline of astronomical observation data. In Star Flow, we will tackle three major components in the processing pipeline: (1) source extraction, (2) image subtraction, and (3) cross match. In source extraction, the tabular attributes of hundreds of thousands of stars and galaxies are extracted from each observation image. Image subtraction and cross match are two alternatives of finding transient objects with the difference being whether the input are observation images or tabular data.



Software License

The license is a free non-exclusive, non-transferable license to reproduce, use, modify and display the source code version of the Software, with or without modifications solely for non-commercial research, educational or evaluation purposes. The license does not entitle Licensee to technical support, telephone assistance, enhancements or updates to the Software. All rights, title to and ownership interest in Software, including all intellectual property rights therein shall remain in HKUST.


We thank our collaborator the National Astronomical Observatories of China for providing us application requirements, access to their data sets, and sharing their domain knowledge. Funding for this project is provided by grants 616012, 617509, and 16206414 from the Hong Kong Research Grants Council and MRA11EG01 from Microsoft SQL Server China R&D.