Towards workload-driven adaptation of data organization in heterogeneous storage systems

Abstract

Collecting, managing, and analyzing huge data sets (Big Data) in science and industry poses challenges to current data storage systems in terms of storage capacity, performance, and reliability. In particular, the I/O performance may be a key factor to speed up the data analysis. However, the performance of a storage system significantly depends on its configuration and the access pattern. Designing storage systems always implies making compromises between performance, fault tolerance and net capacity. The decision which compromise is made (which RAID level is used) has to be taken at deployment time because runtime reconfigurations are usually prohibitively expensive (due to coarse granularity) in current storage architectures. In this paper, we propose a workload-driven approach to adaptive reconfiguration covering the functionality of the file system, volume manager and RAID. Our approach enables fine-grained reconfigurations of the data organization of files and file fragments to adapt the storage system to changing workloads, while considering the different characteristics of the storage devices (SSDs and HDDs) in a heterogeneous storage system. We first discuss how our approach decreases the costs of adaptations compared to existing approaches making a continuous and effective adaptation feasible, even for large volumes of data. Then, we present an evaluation based on a prototypical implementation confirming the benefits of our approach. © 2014 Springer-Verlag Berlin Heidelberg.