DObjects: Enabling distributed data services for metacomputing platforms

Abstract

Many applications rely heavily on large amounts of data in the distributed storages collected over time or produced by large scale scientific experiments or simulations. The key constraints for building a distributed data query infrastructure for such applications are: scalability, consistency, heterogeneity and network and resource dynamics. We designed and developed DObjects, a general-purpose query and data operations infrastructure that can be integrated with metacomputing middleware. This paper describes the architecture of our data services and shows how those services were integrated with the metacomputing framework offering users an open platform for building distributed applications that require access to data integrated from multiple distributed data sources. © 2008 Springer-Verlag Berlin Heidelberg.