Evolutionary computation access on incremental map reduce for mining large scale data

Abstract

In recent era, data updates arrive constantly from different areas like social network, finance, healthcare, e-commerce etc… Hence the data becomes large and computation on it becomes difficult. A framework for mining data earlyand to refresh the computed result with the new data arrival is proposed. The framework includes an incremental mapreduce method on hadoop with evolutionary computation algorithm for reduction in time complexity and increased accuracy. Proposed approach is a key pair level incremental iterative processing to Mapreduce for mining big data and uses particle swarm optimization to avoid re-computation from scratch on the new data arrived. Thereby the I/O overhead gets reduced for accessing predefined states. Experimental results were tested on three iterative algorithms in hadoop showed good performance compared to traditional mapreduce with sequential computation access.