Reducing Stage Weight Estimation Error of Slow Task Detection in MapReduce Scheduling

Abstract

Hadoop architecture mainly comprises of Hadoop MapReduce and Hadoop Distributed File System (HDFS), for processing big data sets. Distributed processing has been widely used for handling large scale data sets. In the recent years, the volume of data has been increasing exponentially and the scalability of processes is growing too. This is the reason, Hadoop architecture attracted and has been adopted by many cloud computing enterprises. MapReduce is a programming model, created and utilized effectively by Google for performing computations on its large volume data sets. LATE, SAMR and ESAMR scheduling algorithm were all introduced for improving the speculative re-execution of slow tasks over Hadoop’s default job scheduler. In our work, we propose the replacement of k-means used in ESAMR algorithm for task’s stage weights estimation by multilayered, feedforward, non-linear sigmoid perceptron model of Artificial Neural Network, thus improving the efficiency of ESAMR algorithm.