Performance comparison of distributed processing of large volume of data on top of xen and Docker-Based virtual clusters

Abstract

Recently, with the advent of cloud computing, it becomes essential to run distributed computing tasks, such as Hadoop MapReduce tasks, on top of virtual computing nodes instead of physical computing nodes. But, distributed big data processing on top of virtual machines usually causes unbalanced use of physical resources, such as memory, disk I/O and network resources, thus resulting in severe performance problems. In this paper, we show how virtualization methods affect distributed processing of very large volume of data, by comparing Hadoop MapReduce processing performance on top of Xen-based virtual clusters versus Docker-based virtual clusters. In our experiments, we compare the performance of two different virtual clusters by changing virtualization methods, block sizes and node numbers. Our results show that, in terms of the distributed big data processing performance, Docker-based virtual cluster is usually faster than Xen-based virtual cluster, but there exist some cases where Xen is faster than Docker according to the parameters, such as block size and virtual node numbers.