Footprinting Parallel I/O – Machine Learning to Classify Application’s I/O Behavior

Abstract

It is not uncommon to run tens of thousands of parallel jobs on large HPC systems. The amount of data collected by monitoring systems on such systems is immense. Checking each job individually by hand, e.g., for identification of high workloads or detection of anomalies, is infeasible. Therefore, we are looking for an automated approach. Many automated approaches are looking at job statistics over the entire job run time. Information about different activities during the job execution is lost. In our work, we partition the collected monitoring data for each job into a sequence of smaller windows for which we analyze the I/O behavior. Then, we convert the sequence to a footprint vector, where each element shows how often this behavior occurs. After that, the footprint dataset is classified to identify applications with similar I/O behavior. The classes are interpreted by a human which is the only non-automatic step in the workflow. The contribution of this paper is a data reduction technique for monitoring data and an automated job classification method.