recon -- Web and network friendly simulation data formats

Abstract

There are many different commonly used file formats for storing time series data. Most of these file formats are designed with the assumption that the file itself will be locally available to the software that will be reading or writing the data stored in them. While this assumption is an excellent one for desktop based tools with direct access to disk drives capable of moving virtually instantaneously around from sector to sector, there are a growing number of applications for which local access is not necessarily available. For these applications, we’ve initiated the recon project to develop more suitable formats. With the emergence of web and cloud based modeling and simulation technologies, the time has come to explore file formats specifically optimized for nondesktop applications. In this paper, we present a new set of file formats that are specifically designed for web and cloud based approaches. This paper reviews the key requirements for web and cloud enabled applications and then presents a specification for two file formats that address those requirements. When considering the various use cases that drove our requirements, we recognized that two different file formats were actually required. The first format, the wall format, is optimized for writing. The other format, the meld format, is optimized for reading over a network (i.e., minimizing the number of reads and bytes read). We will discuss the layout of each of these formats and describe the use cases for which they are most appropriate. In the open tradition of the Modelica Association, the authors have made specifications and implementations for these formats available as open source libraries with the hope that they will benefit the community as a whole.