Data System and Data Management in a Federation of HPC/Cloud Centers

Abstract

This chapter describes the federation of high-performance computing/data centers by a highly reliable distributed data infrastructure (DDI) as part of the LEXIS project (Large-Scale EXecution for Industry and Society, H2020 GA 825532). LEXIS offers user-friendly, cross-site orchestration for simulation and big data workflows, as well as data management, currently implemented at the IT4Innovations National Supercomputing Center (IT4I, CZ) and the Leibniz Supercomputing Centre (LRZ, DE). The chapter outlines requirements and implementation for such an infrastructure in the European context, considering the FAIR principles of modern research data management. Our high-reliability setup of iRODS (Integrated Rule-Oriented Data System) as data middleware is described, as well as the hardware behind the DDI, including burst buffers. A unified user access is ensured using the LEXIS authentication and authorization infrastructure. The LEXIS DDI has been integrated successfully into the LEXIS platform via REST APIs, and into the European data management landscape via EUDAT interfaces and tools.