Software for Portable Scientific Data Management

Abstract

T he problem of processing large amounts of scientific data is a multifaceted one, and many different groups are addressing parts of the problem. Observational data are pouring in from various data-gathering networks and instruments. Numerical simulations generate vast amounts of data. As the processing power of computer systems increases, the problems of getting large volumes of data into and out of applications and of visualizing the results have become significant obstacles to gaining insight from the data. A related problem is how to organize data sets with a view to the various kinds of access that applications demand. Often a diverse range of computer systems is used for processing related data. A numerical simulation may be set up on one workstation, run on a large parallel computer , and the results visualized on a different graphics workstation. Some or all of these computers may have different binary data representations. This presents its own set of problems. In this article we discuss the general philosophy of portable scientific-data-management systems and compare four of the systems that address these kinds of problems: CDF, HDF, netCDF, and PDB. Introduction Traditional methods for handling scientific data include the use of flat sequential files of machine-specific binary data or portable ASCII data. For applications such as visualization, such methods are inefficient in storage, access, and ease-of-use. Relational database systems fail to accommodate the multidimensional or hierarchical structures often found in scientificdata sets. In addition, relational systems do not provide adequate performance for the size, complexity, and type of access required for many scientific data sets. Object-oriented database