Application-aware storage strategy for scientific data

Abstract

The explosive growth of modern scientific data opens new challenges for storing and accessing very large (petabytes) scale data. Traditional file systems and databases cannot meet the requirements of managing scientific data. Arrays are considered as a natural data model for scientific data. Some science-oriented systems have been developed for array data model handling. However, a shortcoming of those systems is that most of them use a "no overwrite" storage strategy, which destabilizes the performance of supporting different applications. In this paper, we proposed an application-aware storage strategy which can optimize data layout gradually according to different access patterns. We implemented the strategy based off of SciDB by creating arrays with different indices for specific parts of the dataset. Experiment testing has been conducted to verify the proposed strategy, and the experimental results show that our strategy improves the performance of science-oriented database on supporting various kinds of applications. © Springer-Verlag Berlin Heidelberg 2012.