Dealing with the big data - The challenges for modern mission monitoring and reporting

Abstract

EUMETSAT maintains a combination of proprietary and commercial-off-the-shelf software, along with supporting infrastructure in order to perform monitoring and reporting of mission performance. The monitoring and reporting falls into three main categories-periodic system monitoring, on-event log file monitoring, and closed loop end-to-end data flow monitoring, which is used not only for near-real time monitoring but also as a basis for operational performance reporting of Key Performance Indicators (KPIs). This system has grown over the past 15 years from processing less than a thousand log file entries per day up to a projected 80 million entries with the advent of upcoming Satellite Programmes-EUMETSAT Polar System (EPS) Second Generation, Jason-CS/Sentinel-6 and Meteosat Third Generation. The combined amount of data from these programmes falls into the big data category and agile solutions must be found in order to provide sufficient processing, storage, and retrieval capability. This paper explores the challenges and potential solutions of preparing for big data in the mission performance monitoring systems, including the software and technology stacks used, methods such as pre-processing data to allow quick access to KPIs, and an scalable infrastructure allowing for easy migration to more performant hardware through the use of virtualized environments. In recent years the technology stack as well as the architectural design for the proprietary monitoring and reporting tools have been evolved in readiness for big data. Storage of data is now performed using a Relational Database Management System, with a Java processing layer contained within a web server. The default method of event persistence is now based on Message Queue (MQ) technology, however other methods of persistence are available such as HTTP RESTful and legacy FTP to ensure compatibility. One major consideration for a big data system is the ability to aggregate data in order to report on KPIs. In order to facilitate this, reporting data is pre-processed and aggregated on a daily basis, meaning the underlying data does not need to be accessed for common reports. For ad-hoc style reporting, there is an option to run reports as a background task with a notification to the user upon completion. The reporting made available to the users through an intuitive web interface, including options to subscribe to report delivery via email. In terms of hardware, the platform used to support the mission performance monitoring has been upgraded as a matter of obsolescence replacement to a virtualized environment based on Hypervisors. While this current generation of hardware is only sufficient for the current missions, moving to a virtualized environment ensures that migration to more performance hardware can be performed easily and in a way that is largely transparent to the users. While scalability on the hardware level is therefore ensured, and the current software design and technology stack fulfils the current requirements and projected data sizing, there are still concerns regarding software performance in accessing such large amounts of information. Therefore alternative COTS solutions for the proprietary software are also being considered with a potential for hybrid solutions based on proven and reliable technologies. In addition, it is intended to re-analyse the overall approach for mission performance monitoring and reporting along with an industry and best-practice study to understand the most efficient and effective methods of meeting user needs for mission performance monitoring and reporting.