The File Mover: High-performance data transfer for the grid

Abstract

The exploration in many scientific disciplines (e.g. High-Energy Physics, Climate Modeling, and Life Sciences) involves the production and the analysis of massive data collections, whose archival, retrieval, and analysis require the coordinated usage of high-capacity computing, network, and storage resources. To obtain satisfactory performance, these applications require the availability of a high-performance, reliable data transfer mechanisms, able to minimize the data transfer time that often dominates their execution time. In this paper we present the File Mover, an efficient data transfer system specifically tailored to the needs of data-intensive applications, that exploits the overlay networks paradigm to provide superior performance with respect to conventional file transfer systems. An extensive experimental evaluation, carried out by means of a proof-of-concept implementation of the File Mover for a variety of network scenarios, shows the ability of the File Mover to outperform alternative data transfer systems. Copyright © 2007 John Wiley & Sons, Ltd.