An efficient approach for mapreduce result verification

Abstract

Hadoop follows a master-slave architecture and can process massive amount of data by using the MapReduce paradigm. The major problem associated with MapReduce is correctness of the results generated. Results can be altered and become wrong by the collaboration of malicious slave nodes. Credibility-based result verification is one of the effective methods to determine such malicious nodes and wrong results. The major limitation of the approach is that, it depends on the complete results of long-running jobs to identify malicious nodes and hence holds valuable resources. In this paper, we propose a new protocol called Intermediate Result Collection and Verification (IRCV) Protocol that prunes out unnecessary computations by collecting results for verification earlier in the execution line. In addition, unlike the previous approach, IRCV uses only a subset of nodes for the purpose. Our simulation experiments suggest that the new approach has improved performance and will lead to better utilization of resources.