In-database geospatial analytics using python

Abstract

The amount of spatial data acquired from crowdsourced platforms, mobile devices, sensors and cartographic agencies has grown exponentially over the past few years. Nearly half of the spatial data available currently are stored and processed through large relational databases. Due to a lack of generic open source tools, researchers and analysts often have difficulty in extracting and analyzing large amounts of spatial data from traditional databases. In order to overcome this challenge, the most effective way is to perform the analysis directly in the database, which enables quick retrieval and visualization of spatial data stored in relational databases. Also, working in-database reduces the network overhead, as users do not need to replicate the complete data into their local system. While a number of spatial analysis libraries are readily available, they do not work in-database, and typically require additional platform-specific software. Our goal is to bridge this gap by developing a new method through an open source software to perform fast and seamless spatial analysis without having to store the data in-memory. We propose a framework implemented in Python, which embeds geospatial analytics into a spatial database (i.e. IBM DB2 ®). The framework internally translates the spatial functions written by the user into SQL queries, which follow the standards of Open Geospatial Consortium (OGC) and can operate on single as well as multiple geometries. We then demonstrate how to combine the results of spatial operations with visualization methods such as choropleth maps within Jupyter notebooks. Finally, we elaborate upon the benefits of our approach via a real-world use case, in which we analyze crime hotspots in New York City using the in-database spatial functions.