Fast approximate attribute reduction with MapReduce

Abstract

Massive data processing is a challenging problem in the age of big data. Traditional attribute reduction algorithms are generally time-consuming when facing massive data. For fast processing, we introduce a parallel fast approximate attribute reduction algorithm with MapReduce. We divide the original data into many small blocks, and use reduction algorithm for each block. The reduction algorithm is based on attribute significance. We compute the dependency of each reduction on testing data in order to select the best reduction. Data with different sizes are experimented. The experimental results show that our proposed algorithm can efficiently process large-scale data on Hadoop platform. In particular, on high dimensional data, the algorithm runs significantly faster than other latest parallel reduction methods. © 2013 Springer-Verlag.