Commutativity of reducers

Abstract

In the Map-Reduce programming model for data parallel computation, a reducer computes an output from a list of input values associated with a key. The inputs however may not arrive at a reducer in a fixed order due to nondeterminism in transmitting key-value pairs over the network. This gives rise to the reducer commutativity problem, that is, is the reducer computation independent of the order of its inputs? In this paper, we study the reducer commutativity problem formally.We introduce a syntactic subset of integer programs termed integer reducers to model real-world reducers. In spite of syntactic restrictions, we show that checking commutativity of integer reducers over unbounded lists of exact integers is undecidable. It remains undecidable even with input lists of a fixed length. The problem however becomes decidable for reducers over unbounded input lists of bounded integers.We propose an efficient reduction of commutativity checking to conventional assertion checking and report experimental results using various off-the-shelf program analyzers.