Translation of array-based loops to distributed data-parallel programs

Abstract

Large volumes of data generated by scientific experiments and simulations come in the form of arrays, while programs that analyze these data are frequently expressed in terms of array operations in an imperative, loop-based language. But, as datasets grow larger, new frameworks in distributed Big Data analytics have become essential tools to large-scale scientific computing. Scientists, who are typically comfort-able with numerical analysis tools but are not familiar with the intricacies of Big Data analytics, must now learn to con-vert their loop-based programs to distributed data-parallel programs. We present a novel framework for translating programs expressed as array-based loops to distributed data parallel programs that is more general and ecient than re-lated work. We report on a prototype implementation on top of Spark and evaluate the performance of our system relative to hand-written programs.