Adaptable, Metadata Rich IO Methods for Portable
High Performance IO
Jay Lofstead
College of Computing
Georgia Institute of Technology
Atlanta, Georgia
lofstead@cc.gatech.edu
Fang Zheng
College of Computing
Georgia Institute of Technology
Atlanta, Georgia
fzheng8@mail.gatech.edu
Scott Klasky
Oak Ridge National Laboratory
Oak Ridge, Tennessee
klasky@ornl.gov
Karsten Schwan
College of Computing
Georgia Institute of Technology
Atlanta, Georgia
schwan@cc.gatech.edu
Abstract—Since IO performance on HPC machines strongly
depends on machine characteristics and configuration, it is
important to carefully tune IO libraries and make good use
of appropriate library APIs. For instance, on current petascale
machines, independent IO tends to outperform collective IO, in
part due to bottlenecks at the metadata server. The problem
is exacerbated by scaling issues, since each IO library scales
differently on each machine, and typically, operates efficiently to
different levels of scaling on different machines. With scientific
codes being run on a variety of HPC resources, efficient code
execution requires us to address three important issues: (1) end
users should be able to select the most efficient IO methods
for their codes, with minimal effort in terms of code updates
or alterations; (2) such performance-driven choices should not
prevent data from being stored in the desired file formats, since
those are crucial for later data analysis; and (3) it is important
to have efficient ways of identifying and selecting certain data for
analysis, to help end users cope with the flood of data produced
by high end codes. This paper employs ADIOS, the ADaptable
IO System, as an IO API to address (1)-(3) above. Concerning (1),
ADIOS makes it possible to independently select the IO methods
being used by each grouping of data in an application, so that end
users can use those IO methods that exhibit best performance
based on both IO patterns and the underlying hardware. In
this paper, we also use this facility of ADIOS to experimentally
evaluate on petascale machines alternative methods for high
performance IO. Specific examples studied include methods that
use strong file consistency vs. delayed parallel data consistency,
as that provided by MPI-IO or POSIX IO. Concerning (2), to
avoid linking IO methods to specific file formats and attain high
IO performance, ADIOS introduces an efficient intermediate file
format, termed BP, which can be converted, at small cost, to the
standard file formats used by analysis tools, such as NetCDF and
HDF-5. Concerning (3), associated with BP are efficient methods
for data characterization, which compute attributes that can be
used to identify data sets without having to inspect or analyze
the entire data contents of large files.
I. INTRODUCTION
File formats like HDF and NetCDF are popular in part
due to the rich tool chains available for the scientific data
stored using these formats. Both HDF and NetCDF, however,
were initially designed for serial access, limiting scalability
when used in massively parallel codes. In response, the
broader community has developed parallel versions of their
APIs [1], [2], with good results demonstrated in the terascale
environment compared to their serial counterparts. However,
there remain serious scalability issues for petascale machines
and beyond, a simple example explained in this paper being the
inability of HDF-5 to scale to 8192 processes for benchmarks
conducted with the Chimera supernova code. Here, with every
performance option enabled for parallel HDF-5 enabled, we
measure 1400 seconds to write a 7 GB restart file to the
Lustre system on the Jaguar machine at Oak Ridge National
Laboratories (ORNL), whereas the use of our ADaptable IO
System (ADIOS) and its IO API with underlying POSIX IO
reduces that time to 1.4 seconds! ADIOS with MPI-IO and
collective MPI-IO yields performance of 10 and 14 seconds,
respectively. BP conversion to HDF-5 performed serially on a
single login node on Jaguar requires 117 seconds.
IO performance depends on many factors, including the file
format used, the implementation and tuning of the associated
API, the file system employed, and the architecture of the
HPC resource being used for production runs. Our analysis
of the parallel HDF-5 implementation on Jaguar, for instance,
reveals a large number of MPI Bcast calls, which are used to
guarantee that all processes writing the collective values are
writing consistent data to the file and that all IO processes
maintain a coordinated march through the data elements for
each collective output. We conclude from facts like these that
ADIOS should provide end users easy access to multiple IO
methods. On Jaguar, for instance, given the severe performance
impact of validating the parallel consistency of data output,
end users might use HDF-5 and Lustre during testing, but then
disable consistency checking during large-scale production
runs. Unfortunately, the parallel HDF-5 and parallel NetCDF
APIs do not offer this capability, since both the HDF-5 and
NetCDF file formats are designed to provide a single view
of the entire data in the file. The laudable idea is that file
contents should not change with say, 30,000 processes all