CNES Gaia Data Processing Centre, a complex operation plan

Abstract

CNES is in charge of a large part of the Gaia data processing. The Gaia astronomy mission aims at mapping one billion stars and objects. The Gaia project, the Data Processing and Analysis Consortium and the role of CNES are presented. The satellite is planned to be launched in 2013.This paper focus on the complexity of the operation plan in terms of data volume, of processing systems and of organisation as scientific algorithm development will continue during operations. Gaia scientific data processing will be daily and cyclic, cycles planned to last from 4 months to one year. Two daily and 5 cyclic large scientific chains will run in the CNES Data Processing Centre (DPCC). Due to the iterative data reduction, the number of systems in operation and the data volumes will increase at each cycle. The operation plan required to obtain the final stars catalogue will be presented. The data volumes will go from some TeraBytes at the beginning of the mission to PetaBytes at the end. To deal with this we have chosen an Hadoop solution that is highly scalable in terms of data volume and processing. We will present the operation framework that will enable to run the operations. Some scientific systems will come into operations more than two years after launch so they will not be ready at launch time and development will go on. We will present the organisation planned to deal with this. © 2012 by CNES.