Data Discoverability in Science Gateways at Scale using Elasticsearch Cluster Architecture

Abstract

Science gateways allow science & engineering communities to access shared data, software, computing services, instruments, educational materials, and other resources specific to their disciplines. One specific example is the use of science gateways to connect researchers with HPC resources by providing a graphical interface to submit jobs and manage shared data sets. In addition to job and data management, the ability to offer robust search features are highly valuable additions to gateways because they enhance the navigability of a user's personal data as well as the discoverability of collaborative data resources. For a facility managing multiple science gateway products, maintaining up-to-date search indices is a challenge. In this paper, we discuss our framework, architecture, and operation of a multitenant Elasticsearch cluster designed to fulfill the search needs of an expanding portfolio of science gateways. By leveraging Elasticsearch's distributed data model and role-based access control, we designed a secure search solution which has scaled to over 200 million indexed entities representing approximately 700 terabytes of research data.