Do faster routers imply faster communication?

Abstract

Despite significant improvements in network interfaces and software messaging layers, software communication overhead still dominates the hardware routing cost in most parallel systems. In this study, we identify the sources of this overhead by relating user communication services to particular network hardware features. Based on a detailed analysis of the active messages layer on the CM-5, we assign the software messaging cost to specific user communication services and network features. Our study shows that 50–70% of the software cost of messaging can be attributed to providing end-to-end flow control, in-order delivery, and reliable transmission services. This overhead is a direct effect of specific network features — arbitrary delivery order, finite buffering, and limited fault-handling — and is unlikely to be eliminated through improved software implementations. We conclude that reducing this software overhead requires changing the constraints on messaging layers: we propose designing routing networks which replace (or make unnecessary) software to implement user services. Specifically, networks which provide ordering, end-to-end flow control, and hardware-supported fault-tolerance can significantly reduce the end-to-end cost of communication.