Improving parallel I/O performance using multithreaded two-phase I/O with processor affinity management

Abstract

I/O has been one of the performance bottlenecks in parallel computing. Using a parallel I/O API such as MPI-IO is one effective approach to improve parallel computing performance. The most popular MPI-IO implementation, ROMIO, utilizes two-phase I/O technique for collective I/O for non-contiguous access patterns. Furthermore, such two-phase I/O is frequently used in application oriented parallel I/O libraries such as HDF5 through an MPI-IO interface layer. Therefore performance improvement in the two-phase I/O may have a big impact in improving I/O performance in parallel computing. We report enhancements of the two-phase I/O by using Pthreads in order to improve I/O performance in this paper. The enhancements include overlapping scheme between file I/O and data exchanges by multithreaded operations and the processor affinity for threads dedicated for file I/O and data exchanges. We show performance advantages of the optimized two-phase I/O with an appropriate processor affinity management relative to the original two-phase I/O in parallel I/O throughput evaluation of HDF5. © 2014 Springer-Verlag.