Realizing the performance potential of the Virtual Interface Architecture

Abstract

The Virtual Interface (VI) Architecture provides protected user-level communication with high delivered bandwidth and low per-message latency, particularly for small messages. The VI Architecture attempts to reduce latency by eliminating user/kernel transitions on routine data transfers and by allowing direct use of user memory for network buffering. This results in significantly lower latencies than those achieved by network protocols such as TCP/IP and UDP. In this paper we examine the low-level performance of two VI implementations, one implemented in hardware, the other implemented in device driver software. Using a set of low-level benchmarks, we measure bandwidth, latency, and processor utilization as a function of message size for the GigaNet cLAN and Tandem ServerNet VI implementations. We report that both VI implementations offer significant performance advantage relative to the corresponding UDP implementation on the same hardware. We also investigate the problems associated with delivering this performance to distributed applications running on clustered multiprocessor workstations. Using an existing MPI library implemented using UDP as a baseline, we explore several performance and implementation issues that arise when adapting this library to use VI instead of UDP. Among these issues are memory registration costs, polling vs. blocking, reliability of delivery, and memory-to-memory copying. By eliminating explicit acknowledgements, reducing memory-to-memory copying, and choosing the most appropriate synchronization primitives, we reduced the message latency seen by user applications by an average of 55% across all message sizes. We also identify several areas that offer the potential for further performance enhancement.