A processing pipeline for high volume pulsar candidate data streams

Abstract

Pulsar candidate analysis pipelines have historically been comprised of bespoke software systems, supporting the off-line study of data. However modern data acquisition systems are making off-line analyses impractical, as increasing candidate volumes become prohibitively expensive to retain. To maintain processing capabilities when off-line analysis becomes infeasible due to cost, requires a shift to on-line data processing. This paper makes four contributions facilitating this shift relevant to the search for radio pulsars: (i) it characterises the key components of a pulsar search candidate processing pipeline, (ii) it examines the feasibility of implementing on-line candidate filtering via existing tools, (iii) explores the problems preventing an easy transition to on-line filtering, and finally (iv) presents a new prototype filtering pipeline capable of overcoming such problems. Realised using Commercial off-the-shelf (COTS) software components, the deployable system is open source, simple, scalable, and cheap to produce. It has the potential to achieve candidate filtering design requirements for the Square Kilometre Array (SKA), illustrated via testing under simulated SKA loads.