Advancing privacy-aware machine learning on sensitive data via edge-based continual µ-training for personalized large models

Abstract

This paper introduces an innovative method for fine-tuning a large multi-label model for abnormality detection, utilizing a smaller trainer and advanced knowledge distillation techniques. It studies the effects of fine-tuning on various abnormalities, noting different improvements based on the Original Model’s performance in specific tasks. The experimental setup, optimized for on-device inference and fine-tuning with limited computational resources, demonstrates moderate yet promising enhancements in model performance post-fine-tuning. Key insights from the study include the significance of aligning the µ-Trainer’s behavior with the Original Model and the influence of hyper-parameters like batch size on fine-tuning outcomes. The research acknowledges limitations such as the limited exploration of loss functions in multi-label models and constraints in architectural design, suggesting potential avenues for future investigation. While the proposed Naive Continual Fine-tuning Process is in its early stages, we highlight this paper’s potential model personalization on long-term data. Moreover, weight transfer in our system is exclusively for fine-tuning; hence, it improves user privacy protection by failing data reconstruction attempts from weights, like an issue with Federated learning models. Our on-device fine-tuning prevents the transferring of data or gradients from the edge of the network to their server. Despite modest performance improvements after fine-tuning, these working layers represent a small fraction (0.7%) of the total weights in the Original Model and 1.6% in the µ-Trainer. This study establishes a foundational framework for advancing personalized model adaptation, on-device inference and fine-tuning while emphasizing the importance of safeguarding data privacy in model development.