Leveraging Gradient Noise for Detection and Filtering of Byzantine Clients

Abstract

Distributed Learning enables multiple clients to collaboratively train large models on private, decentralized data. However, this setting faces a significant challenge: real-world datasets are inherently heterogeneous, and the distributed nature of the system makes it vulnerable to Byzantine attacks. This combination makes it difficult for standard aggregations to reliably distinguish between honest clients with atypical data and malicious participants attempting to disrupt the training. To address this problem, we propose a novel Byzantine defense that leverages the statistical behavior of stochastic gradient noise (GN) in deep learning to identify malicious clients. Unlike pre-processing techniques that attempt to reduce inter-client gradient variance, our method directly exploits inherent gradient properties to filter malicious updates. Our aggregation can be deployed as a standalone defense or in combination with existing robust aggregation rules. We provide theoretical guarantees on convergence under standard assumptions. Our empirical evaluations on different benchmark datasets further demonstrates that our approach achieves high detection accuracy across a range of attack scenarios, significantly improving robustness without sacrificing model performance.