A data cleaning service on massive spatio-temporal data in highway domain

Abstract

With the development of highway toll system and sensor network, massive highway toll data has been accumulated nowadays. The imperfection of raw data, such as incomplete, repetitive and abnormal data, seriously affects the efficiency of data mining modeling. Traditional cleaning methods on massive spatio-temporal data are inefficient, because the business rules are difficult to depict in various domains. On the highway toll data of Henan Province, we propose a data cleaning service through business rules. This service can efficiently clean the raw toll data with spatio-temporal attributes, including the data calibration of erroneous data and invalid data, the repair of erroneous data, and the filtering of duplicate data. Implemented through Hadoop MapReduce on toll data in highway domain, our service shows its efficiency, accuracy and scalability in extensive experiments.