The M³ massive movement model: a distributed incrementally updatable solution for big movement data exploration

Abstract

Exploratory analysis is an important tool to formulate hypotheses about data and build data-driven models. To efficiently explore massive movement datasets, researcher and analysts require appropriate exploratory analysis tools. However, there is a lack of appropriate tools for movement data exploration that can handle large data volumes. We therefore propose a novel scalable distributed exploratory analysis model for massive movement datasets with billions of records which we call (Formula presented.). (Formula presented.) is more flexible than classical aggregation approaches that use grids with aggregate statistics and it can be updated incrementally with large amounts of data. We demonstrate this new model and its implementation in Apache Spark using massive ship and vehicle movement data with up to 3.9 billion records.