Filter-embedding semiring fusion for programming with MapReduce

Abstract

We show that MapReduce, the de facto standard for large scale data-intensive parallel programming, can be equipped with a programming theory in calculational form. By integrating the generate-and-test programming paradigm and semirings for aggregation of results, we propose a novel parallel programming framework forMapReduce. The framework consists of two important calculation theorems: the shortcut fusion theorem of semiring homomorphisms bridges the gap between specifications and efficient implementations, and the filterembedding theorem helps to develop parallel programs in a systematic and incremental way. © 2012 BCS.