STARE-based Integrative Analysis of Diverse Data Using Dask Parallel Programming Demo Paper

Abstract

Scaling up volume and variety in Big Earth Science Data is particularly difficult when combining low-level, ungridded data, such as swath observations obtained with, for example, Moderate Resolution Imaging Spectroradiometers (MODIS). A unified way to index and combine data with different geo-spatiotemporal layouts and incomparable native array formatting is required for scalable integrative analyses based on data at its full instrument resolution, that is, without extra interpolation (or extrapolation) onto a common grid. The SpatioTemporal Adaptive Resolution Encoding (STARE) uses the Hierarchical Triangular Mesh (HTM) and the Hierarchical Calendrical Partitioning (HCP), recursive partitionings of solid angle and time into tree data structures, to encode spatiotemporal neighborhoods as sets of integers. Regions sharing common paths through the STARE tree hierarchy have similar index values, which can then serve as keys in algorithms and data structures supporting scalable integrative analyses. Thus, STARE co-aligns data in both physical (spatiotemporal) and cyber (memory) spaces, providing a means for marshalling computing resources and conducting analysis with minimum data movement, addressing volume scalability while simultaneously unifying diverse data for variety scaling. In this paper, we demonstrate how easy it is to use the Python STARE API (PySTARE) and the parallel programming platform Dask to integrate MODIS and Geostationary Operational Environmental Satellite (GOES) data, datasets with very different geo-spatiotemporal characteristics.