A software architecture for parallel list processing on grids

Abstract

The Data List Management Library (DLML) processes data lists in parallel, balancing the workload transparently to programmers. Programmers only need to organise data into a list, use DLML functions to insert and get data items, and specify the sequential function(s) to process each data item according to the application logic. The first design of DLML was targeted for use at a single cluster. This paper presents DLML-Grid, a software architecture for DLML to run in Grid environments composed of multiple distributed clusters. The architecture is hierarchical and tends to localise communication within clusters, thus reducing communication overhead. Using OpenVPN, we implemented a prototype version of DLML-Grid to gather empirical results on its performance using two clusters and two applications whose workload is static and dynamically generated. DLML-Grid performs much better than DLML overall. © 2012 Springer-Verlag.