Scalable MapReduce framework on FPGA accelerated commodity hardware

Abstract

Running MapReduce framework for massive data processing on a cluster of commodity hardware requires enormous resource, especially high CPU and memory occupation. To enhance the commodity hardware performance without physical update and topology change, the highly parallel and dynamically configurable FPGA can be dedicated to provide feasible supplements in computation running as coprocessor to CPU. This paper presents a MapReduce Framework on FPGA accelerated commodity hardware. In our framework, a cluster of worker nodes is designed for MapReduce framework, and each worker node consists of commodity hardware and special hardware. CPU base worker runs the major communications with other worker node and tasks, while FPGA base worker operates extended mapreduce tasks process to speed up the computation process. Due to internal pipeline in computing operations, FPGA base worker offers the high performance which enhances 10x faster task processing. Furthermore, CPU base worker can reconfigure FPGA chip immediately when it fails. In this period, data will be migrated and continuously processed in commodity hardware. Meanwhile a local memory in commodity hardware is implemented to recover the lost data. Moreover, most frequent computation modules are provided in FPGA module library which are convenient for user to configure operations in special hardware. Experimental results proves that our framework offers high performance and flexibility in applications. © 2012 Springer-Verlag.