Array DBMS and satellite imagery: Towards big raster data in the cloud

Abstract

Satellite imagery have always been “big” data. Array DBMS is one of the tools to streamline raster data processing. However, raster data are usually stored in files, not in databases. Respective command line tools have long been developed to process these files. Most of the tools are feature-rich and free but optimized for a single machine. The approach of partially delegating in situ raster data processing to such tools has been recently proposed. The approach includes a new formal N-d array data model to abstract from the files and the tools as well as new formal distributed algorithms based on the model. ChronosServer is a distributed array DBMS under development into which the approach is being integrated. This paper extends the approach with a new algorithm for the reshaping (tiling) of arbitrary N-d arrays onto a set of overlapping N-d arrays with a fixed shape. Cutting arrays with an overlap enables to perform a broad range of large imagery processing operations in a distributed shared-nothing fashion. Currently ChronosServer provides a rich collection of raster operations at scale and outperforms SciDB up to 80 × on Landsat data. SciDB is the only freely available distributed array DBMS to date. Experiments were carried out on 8- and 16-node clusters in Microsoft Azure Cloud.