Development of a Polystore Data Management System for an Evolving Big Scientific Data Archive

Abstract

Handling large datasets can be a big challenge in case of most astronomical data repositories. Many astronomical repositories manage images, text, key-values, and graphs that make up the enormous volume of data available in the astronomical domain. Palomar Transient Factory (PTF/iPTF) is one such project which has relational data, image data, lightcurve data sets, graphs, and text data. Organizing these data in a single data management system may have low performance and efficiency issue. Thus, we propose to demonstrate a prototype system to manage such heterogeneous data with multiple storage units using polystore based approaches. The prototype supports a set-theoretic query language for access to cloud-based data resources.