Using APSIM, C# and R to create and analyse large datasets

Abstract

With the advent of cheap cluster computing, advanced models and improved analytical techniques, scientists are able to explore larger problem spaces than ever before. However, with this power comes added complexity. Large factorial simulations need to be designed and created then the output - which can be many hundreds of gigabytes or more - needs to be analysed and presented in a meaningful way. This paper details one such scenario using the Agricultural Production Systems Simulator (APSIM) (Holzworth et al, 2014). Almost 60 000 simulations were created using five base simulations that generated approximately 100GB of output data. This output data was loaded into R using a new package designed for loading, testing, manipulating and exporting both input and output APSIM formatted data. This paper explores the mechanics of large scale APSIM simulations including how to leverage the APSIM User Interface (UI) to quickly build a starting point simulation, using the XML libraries in the.NET framework (C# in this case) to easily duplicate, replace and modify structures in the base simulation to create a large factorial and finally looks at methods to analyse large output data sets. The techniques demonstrated here are scalable; while 60 000 simulations were generated for this project, the same method could be applied to a grid analysis with millions of simulations. Similarly, the processing in R will work with any size data set so long as the computer used has enough memory. Even the language used for generating the simulations can be changed. C# is used here but any language with a robust XML interpreter could be used. In order to keep the analysis run time down and simplify factorial combination generation, the top level factor (crop type) was used as a divider to split up the output into individual batches that were then processed in R. APSIM writes a single output file for each simulation. As such, large factorial simulations can produce millions of individual files that need to be processed. To assist with this, an R package has been created and is available on CRAN (under the package name 'APSIM') that automates the loading of multiple APSIM output files into a single R data frame or data table (Dowle, et al., 2014). This package is able to handle single output and factorial simulations as well as import constants as separate columns. Additionally, extra utilities have been added that simplify the process of creating meteorological data files for simulation input. R has many ways of doing the same thing, some of which are more efficient than others. A number of processes for reading and parsing data were evaluated and the most efficient methods were incorporated into the APSIM package. This evaluation has applications beyond APSIM as they are methods that anyone importing large quantities of data into R will be able to implement. The package assumes that any data sets loaded will fit into system memory. As simulations become more complex this assumption will become less valid and new approaches will need to be taken such as the use of databases to store data that is not being actively worked on. There are R packages already in existence that do this and the APSIM package will be expanded over time to utilise these options. Optimisations used include minimising file access, binding files after reading all of them, using dedicated data reading packages and applying techniques that reduce the amount of memory management required. The optimised data reader was able to import and bind APSIM output data at speeds in excess of 11MB/s for 5MB data files.