A framework for statistical analysis of datasets on heterogeneous clusters

Abstract

This paper proposes a framework for the statistical analysis of multiple related datasets on heterogeneous clusters. The set of processors assigned to the framework are partitioned into groups according to rack locations, with the group sizes being chosen to match the degree of concurrency in the analysis procedure. The datasets are initially divided among the groups. Dynamic loop scheduling is employed to address load imbalance arising from the differences in computational powers of groups, the variability of dataset sizes, and the unpredictable irregularities in the cluster environment. Results of preliminary tests indicate the effectiveness of the framework in fitting gamma-ray burst datasets with vector functional coefficient autoregressive time series models on 64 processors of a heterogeneous general-purpose Linux cluster.