SmartIdx: Reducing Communication Cost in Federated Learning by Exploiting the CNNs Structures

Abstract

Top-k sparsification method is popular and powerful for reducing the communication cost in Federated Learning (FL). However, according to our experimental observation, it spends most of the total communication cost on the index of the selected parameters (i.e., their position information), which is inefficient for FL training. To solve this problem, we propose a FL compression algorithm for convolutional neural networks (CNNs), called SmartIdx, by extending the traditional Top-k largest variation selection strategy into the convolution-kernel-based selection, to reduce the proportion of the index in the overall communication cost and thus achieve a high compression ratio. The basic idea of SmartIdx is to improve the 1:1 proportion relationship between the value and index of the parameters to n:1, by regarding the convolution kernel as the basic selecting unit in parameter selection, which can potentially deliver more information to the parameter server under the limited network traffic. To this end, a set of rules are designed for judging which kernel should be selected and the corresponding packaging strategies are also proposed for further improving the compression ratio. Experiments on mainstream CNNs and datasets show that our proposed SmartIdx performs 2.5×-69.2× higher compression ratio than the state-of-the-art FL compression algorithms without degrading training performance.