Evaluating and Enhancing the Generalization Performance of Machine Learning Models for Physical Activity Intensity Prediction from Raw Acceleration Data

Abstract

Purpose: To evaluate and enhance the generalization performance of machine learning physical activity intensity prediction models developed with raw acceleration data on populations monitored by different activity monitors. Method: Five datasets from four studies, each containing only hip-or wrist-based raw acceleration data (two hip-and three wrist-based) were extracted. The five datasets were then used to develop and validate artificial neural networks (ANN) in three setups to classify activity intensity categories (sedentary behavior, light, and moderate-to-vigorous). To examine generalizability, the ANN models were developed using within dataset (leave-one-subject-out) cross validation, and then cross tested to other datasets with different accelerometers. To enhance the models' generalizability, a combination of four of the five datasets was used for training and the fifth dataset for validation. Finally, all the five datasets were merged to develop a single model that is generalizable across the datasets (50%of the subjects from each dataset for training, the remaining for validation). Results: The datasets showed high performance in within dataset cross validation (accuracy 71.9-95.4%, Kappa K = 0.63-0.94). The performance of the within dataset validated models decreased when applied to datasets with different accelerometers (41.2-59.9%, K = 0.21-0.48). The trained models on merged datasets consisting hip and wrist data predicted the left-out dataset with acceptable performance (65.9-83.7%, K = 0.61-0.79). The model trained with all five datasets performed with acceptable performance across the datasets (80.4-90.7%, K = 0.68-0.89). Conclusions: Integrating heterogeneous datasets in training sets seems a viable approach for enhancing the generalization performance of the models. Instead, within dataset validation is not sufficient to understand the models' performance on other populations with different accelerometers.