Tagged MapReduce: Efficiently computing multi-analytics using MapReduce

Abstract

MapReduce is a programming paradigm for effective processing of large datasets in distributed environments, using the map and reduce functions. The map process creates (key, value) pairs, while the reduce phase aggregates same-key values. In other words, a MapReduce application defines and reduces one set of values for each key, which means that the user only knows one aspect of the key. Advanced OLAP applications however, require multiple sets to be defined and reduced for the same key, not necessarily mutually disjoint. The challenge is to extend MapReduce to support this in a syntactically simple and computationally efficient way. We propose an extension to the classic MapReduce model, called Tagged MapReduce, where data is represented as (key, value, tag) triplets. Users map triplets and reducing takes place for each key and for each tag. For example, given a set of pages, one may want to count words' occurrences per page type. The page type is represented by the tag. While the classic MapReduce can handle this class of queries, it requires effort and possibly advanced programming skills for efficient implementations. For example, should the tag form a compound object with the key or the value? Our formalism makes it simpler for the programmer to use and easier for the system to identify and apply efficient algorithms. © 2011 Springer-Verlag.