An Adaptive Parameters Density Cluster Algorithm for Data Cleaning in Big Data

Abstract

We have entered the era of information explosion, data has become an important driving force for the development of the industry. The huge wealth hidden in the data, enterprises can obtain a lot of useful information from business management, market analysis, scientific exploration and other aspects to support the development decisions of enterprises. However, the actual data is often intricate with different structures and types such as erroneous data, invalid data and missing duplicate data, greatly increase the difficulty of data analysis. These dirty data can greatly affect the results of data analysis, resulting in inaccurate results or even bad information. Most of the early data cleaning requires human involvement, however the exploding large-scale data is far from being able to meet with the human intervention, requiring smarter and more automated cleaning methods. The rapid development of artificial intelligence in recent years, the emergence of machine learning, provides a new opportunity for the development of data cleaning. Nowadays, there are lots of methods that have been applied to the field of data mining, including popular machine learning, deep learning, as well as classical clustering, bayesian networks, decision trees, and so on. Both provide a large number of solutions for our data cleaning. In this paper, we present a data cleaning method which use adaptive parameters density cluster algorithms based on the DBSCAN.