Metacomputing and data-intensive applications

Abstract

Metacomputing-the concurrent use of multiple networklinked computers for solving application problems-is gaining increasing popularity. Much of the early work in metacomputing focused on harnessing greater processing power than could be found at a single site or on combining heterogeneous computer architectures to exploit the best features of each for a given problem. Another compelling rationale for using metacomputing is the acquisition, manipulation, and analysis of data that are stored remotely or acquired by distant instruments. When the data to be accessed and processed are voluminous, we refer to the applications as data intensive. Large-scale projects of this kind require leading-edge technologies in multiple fields: high-speed networking, high-performance computing, software for interactive control of programs and instruments, and data storage facilities. This paper describes three Caltech projects that take advantage of metacomputing, are data-intensive, or both. Brief mention is also made of a few additional projects that have these characteristics and are being conducted elsewhere. The applications described include synthetic aperture radar (SAR) data processing; a user interface for controlling the access and processing of scientific data and the use of distributed archives and computers for that data; and large-scale distributed interactive simulations for synthetic theater of war. Since Caltech is not the only institution that is pursuing distributed, data-intensive metacomputing, we also provide examples of interesting projects from other sites, such as telemicroscopy at the University of California at San Diego and the Distributed Object Computation Testbed that is being developed at the San Diego Supercomputer Center. The paper ends with a brief survey of future projects in this area and some developments that will enable the use of metacomputing to more and more distributed, data-intensive applications.