Motivation and implementation of a dynamic remote storage system for I/O demanding HPC applications

Abstract

I/O subsystem performance is becoming increasingly important for a wide range of applications. The demand can be met with a large memory capacity or fast local SSDs, but such solutions cause very high investment costs and are rather inflexible. Here, we investigate a multitiered approach, combining memory, local SSDs and InfiniBand-attached block storage connected via SRP (SCSI RDMA protocol). Different variants of this hybrid storage system are evaluated, and we present a method to analyze the I/O patterns of applications for choosing the best approach. We also demonstrate the integration of the dynamic remote storage facility into the scheduling system Moab. Our method allows for an on-demand provisioning of multi-tiered file systems of varying sizes, managing storage resources automatically with a broker software.