Benchmarking Container Technologies on ARM-Based Edge Devices

Abstract

Internet-of-Things (IoT) devices continuously gather data and send the data to the cloud for further processing. However, with the recent trend of increasing number of IoT devices, billions of devices are anticipated to send data to the cloud, eventually impacting performance and cost. To solve this problem, one way is to process data locally inside edge nodes. The edge nodes are closer to the IoT devices and improve the overall performance of the system by distributing cloud tasks in edge devices. Edge devices are generally resource-constrained with limited RAM, CPU, and storage. Container technologies are ideal in edge nodes due to their isolation and being lightweight. A benchmarking scheme for containers on edge devices can help compare container technologies, hardware devices and architectures, and software. However, there is not sufficient research in this direction. So, in this work, we take a step towards developing this benchmark. We explore and evaluate the performance, efficiency, and suitability of different container technologies, including Docker, Podman, and Singularity, in the context of edge computing on ARM-based devices. Our experiments include evaluating computer vision applications that employ Haar Cascades, HOG, CNN with YOLO algorithm, and data science workloads commonly encountered in edge computing scenarios. We devised sets of performance metrics to assess container technology, including waiting time, receiving time, processing time, resource utilization, and throughput. Besides, we investigate how different container technologies optimize resource utilization and compare their efficiency on ARM-based edge devices. Our benchmarking analysis yielded valuable insights into the strengths and limitations of each container technology. Our results reveal that Docker, Podman, and Singularity containers exhibit diverse resource consumption patterns and network efficiency. Docker container has better CPU and RAM utilization for most applications. Docker also boasts the lowest waiting time of approximately 0.9 seconds, comparable to native performance. In terms of processing time, Docker excels in Car detection (0.12 seconds), while Singularity and Podman outperform Docker in Object detection. Notably, native systems exhibit a remarkable improvement over containers (average of 1.2 seconds) in Object detection (0.98 seconds), highlighting the challenges of resource-intensive deep neural network algorithms on edge devices. Based on our findings, we offer practical advice for picking the best container technology for specific use cases in ARM-based edge computing. We also offer a set of benchmarking approaches and metrics that can be used to drive future research on container technologies on ARM platforms.