A Case Study on Virtual HPC Container Clusters and Machine Learning Applications

Abstract

This article delves into the innovative application of Docker containers as High-Performance-Computing (HPC) environments, presenting the construction and operational efficiency of virtual container clusters. The study primarily focused on the integration of Docker technology in HPC, evaluating its feasibility and performance implications. A portion of the research was devoted to developing a virtual container cluster using Docker. Although the first Docker-enabled HPC studies date back several years, the approach remains highly relevant today, as modern AI-driven science demands portable, reproducible software stacks that can be deployed across heterogeneous, accelerator-rich clusters. Furthermore, the article explores the development of advanced distributed applications, with a special emphasis on Machine Learning (ML) algorithms. Key findings of the study include the successful implementation and operation of a Docker-based cluster. Additionally, the study successfully showcases a Python application using ML for anomaly detection in system logs, highlighting its effective execution in a virtual cluster. This research not only contributes to the understanding of Docker’s potential in distributed environments but also opens avenues for future explorations in the field of containerized HPC solutions and their applications in different areas.