Large-scale Incremental Data Processing with Change Propagation

  • Bhatotia P
  • Wieder A
  • Akku\textbackslashcs \
 et al. 
  • 47

    Readers

    Mendeley users who have this article in their library.
  • N/A

    Citations

    Citations of this article.

Abstract

Incremental processing of large-scale data is an increasingly important problem, given that many processing jobs run repeatedly with similar inputs, and that the de facto standard programmingmodel ({MapReduce)} was not designed to efficiently process small updates. As a result, new systems specifically targeting this problem (e.g., Google Percolator, or Yahoo! {CBP)} have been proposed. Unfortunately, these approaches require the adoption of a new programming model, breaking compatibility with existing programs, and increasing the burden on the programmer, who now is required to devise an incremental update mechanism. We claim that automatic incremental processing of large-scale data is possible by leveraging previous results from the algorithms and programming languages communities. As an example, we describe how Map Reduce can be improved to efficiently handle small input changes by automatically incrementalizing existing {MapReduce} computations, without breaking backward compatibility or demanding programmers to adopt a new programming approach.

Get free article suggestions today

Mendeley saves you time finding and organizing research

Sign up here
Already have an account ?Sign in

Find this document

There are no full text links

Authors

  • Pramod Bhatotia

  • Alexander Wieder

  • \textbackslash.Istemi Ekin Akku\textbackslashcs

  • Rodrigo Rodrigues

  • Umut a Acar

Cite this document

Choose a citation style from the tabs below

Save time finding and organizing research with Mendeley

Sign up for free