ComputErl - Erlang-based framework for many task computing

Abstract

This paper shows how Erlang programming language can be used for creating a framework for distributing and coordinating the execution of many task computing problems. The goals of the proposed solution are (1) to disperse the computation into many tasks, (2) to support multiple well-known computation models (such as master-worker, map-reduce, pipeline), (3) to exploit the advantages of Erlang for developing an efficient and scalable framework and (4) to build a system that can scale from small to large number of tasks with minimum effort. We present the results of work on designing, implementing and testing ComputErl framework. The preliminary experiments with benchmarks as well as real scientific applications show promising scalability on a computing cluster. © 2011 Springer-Verlag.