MEMSCALE: Re-architecting memory resources for clusters

Abstract

Using the widespread and cost-effective cluster computing approach, large amounts of different resources can be aggregated in order to solve compute-intensive problems. However, as clusters do not permit any resource sharing because of their shared-nothing approach, resource partitioning is static. While this is typically not a big problem for computing resources as CPUs, the impact of a static partitioning of memory resources is dramatical. As memory resource shortage leads to demand paging with a huge negative impact on performance, this situation is avoided by overprovisioning of resources. However, due to this provisioning for the worst case, for the average case many memory resources are free but not accessible due to the static partitioning. On the other hand, as shared use of memory resources among different nodes in a cluster is not possible, communication tasks are solved by data replication. MEMSCALE is a new cluster memory architecture that is enabled by a direct low-latency path to remote memory resources. Using MEMSCALE, the static partitioning can be overcome and a dynamic provisioning of resources becomes feasible. Two typical use cases do exist: first, data-intensive applications might benefit from more memory resources, but often not from more computing resources. Such applications can benefit from MEMSCALE by allowing them to borrow memory from remote nodes, rendering the need for overprovisioning obsolete. Therefore, each node can be provisioned for the average case, with significant cost and power savings. The second use case allows applications to rely on shared memory resources for communication and synchronization purposes, avoiding the need for data partitioning and movement. In essence, the shared-memory programming paradigm well known from multi-core computers is expanded to clusters. However, as global continuous coherency is a serious concern when scaling shared-memory systems, we revert to a highly relaxed consistency model, which guarantees consistency only for synchronization intrinsics like barriers and locks. MEMSCALE is implemented using FPGAs in a real cluster prototype. Here, we will use in-memory databases as example workload to demonstrate MEMSCALE's impact for such data-intensive applications.