HPC, Cloud and Big-Data Convergent Architectures: The LEXIS Approach

Abstract

High Performance Computing (HPC) infrastructures (also referred to as supercomputing infrastructures) are at the basis of modern scientific discoveries, and allow engineers to greatly optimize their designs. The large amount of data (Big-Data) to be treated during simulations is pushing HPC managers to introduce more heterogeneity in their architectures, ranging from different processor families to specialized hardware devices (e.g., GPU computing, many-cores, FPGAs). Furthermore, there is also a growing demand for providing access to supercomputing resources as in common public Clouds. All these three elements (i.e., HPC resources, Big-Data, Cloud) make “converged” approaches mandatory to address challenges emerging in scientific and technical domains. The LEXIS project aims to design and set up an innovative computing architecture, where HPC, Cloud and Big-Data solutions are closely integrated to respond to the demands of performance, flexibility and scalability. To this end, the LEXIS architecture leverages on three main distinctive elements: (i) resources of supercomputing centers (geographically located in Europe) which are seamlessly managed in a federated fashion; (ii) an integrated data storage subsystem, which supports Big-Data ingestion and processing; and (iii) a web portal to enable users to easily get access to computing resources and manage their workloads. In addition, the LEXIS architecture will make use of innovative hardware solutions, such as burst buffers and FPGA accelerators, as well as a flexible orchestration software. To demonstrate the capabilities of the devised converged architecture, LEXIS will assess its performance, scalability and flexibility in different contexts. To this end, three computational highly demanding pilot test-beds have been selected as representative of application domains that will take advantage of the advanced LEXIS architecture: (i) Aeronautics – Computational Fluid Dynamics simulations of complex turbo-machinery and gearbox systems; (ii) Earthquake and Tsunami – acceleration of tsunami simulations to enable highly-accurate real-time analysis; and (iii) Weather and Climate – enabling complex workflows which combine various numerical forecasting models, from global & regional weather forecasts to specific socio-economic impact models affecting emergency management (fire & flood), sustainable agriculture and energy production.