Research and application of DBSCAN algorithm based on Hadoop platform

Abstract

Along with the rapid development of information age, more and more data can be obtained from the Internet, it is very difficult to get useful information and knowledge from these huge amounts of data. On the foundation of the existing algorithm based on DBSCAN, a new improved incremental DBSCAN clustering algorithm is proposed. Combining with cloud computing open source framework Hadoop, the improved algorithm use the programming model of MapReduce which can easy write distributed applications and simplify distributed programme to divide a huge amounts of data elements into chunks and distribute the chunks across the cluster and run the algorithm as a MapReduce job, in this way, this improved algorithm of data mining is integrated with framework Hadoop by the DBSCAN clustering algorithm. When data manipulation (add or delete) has occurred in the database, what we need to do is to mine the mutative data and merge the similar clusters, and ultimately form the final knowledge mining.Compared with single node server serial arithmetic and the overall mining, the time delay of data processing will be reduced. In the last part,the paper verified the effectiveness by experiments and data analysis. © 2014 Springer International Publishing.