Optimizing collective I/O performance on parallel computers: a multisystem study

Abstract

While individual parallel I/O systems can incorporate sophisticated techniques and achieve impressive performance in particular situations, researchers as yet have only limited understanding of the impact of various design decisions or of the techniques required for performance robustness. One remedy is to perform detailed comparative studies of different I/O libraries. In this paper, we describe such a study for the Disk Resident Array and Panda libraries, both designed to support high-performance I/O for arrays. While the two systems have many similarities, their designs and implementations are based on different assumptions and target different applications. We base our study on two I/O structures commonly encountered in scientific applications: the collective read/write of an entire array and the collective read/write of an arbitrary array section. Experiments are performed on two parallel file systems (IBM PIOFS and Intel PFS) and one commodity Unix file system (AIX JFS). Our results yield insights into the major parameters determining the performance of array-oriented collective I/O.