Corroborating quality of data through density information

Abstract

Unclean data is a common issue in data quality. Multiple quality issues may exist in data. Over time, data may become obsolete, transcription errors may make data inaccurate, and integration of multiple data sources may cause multiple tuples referring to the same entity to exist in data. Hence, a cleaning process is necessary. Fixing one issue may not clean the data, relying on human effort to correct the data is expensive, and master data or training data may not be available. This paper studies a data cleaning problem by introducing techniques based on corroboration, i.e. taking into consideration the trustworthiness of the attribute values. It presents a data deduplication approach for data that have outdated and inaccurate values in the duplicated tuples. This approach utilizes the density information embedded inside the tuples to guide the cleaning process. This paper introduces a framework and algorithms to integrate data deduplication with data currency and accuracy that can fix multiple data quality issues without the reliance on manual user interaction, master data, or training data.