Optimizing distributed joins with bloom filters using MapReduce

Abstract

The MapReduce framework is increasingly being used to process and analyze large-scale datasets over large clusters. Join operation using MapReduce is an attractive point to which researchers have been paying attention in recent years. The distributed join based on the bloom filter has been proved to be a successful technique to improve the efficiency. However, the full potential of the bloom filter has not been fully exploited, especially in the MapReduce environment. In this paper, we present several strategies to build the bloom filter for the large dataset using MapReduce, compare some bloom-join algorithms and point out how to improve the performance of two-way and multi-way joins. The experiments we conduct show that our method is feasible and effective. © 2012 Springer-Verlag.