Knowledge Distillation Based on Pruned Model

Abstract

The high computational complexity of deep neural networks makes them challenging to deploy in practical applications. Recent efforts mainly involve pruning and compression the weights of layers to reduce these costs, and use randomly initializing weights to fine-tune the pruned model. However, these approaches always lose important weights, resulting in the compressed model performing that is even worse than the original model. To address this problem, we propose a novel method replaced the traditional fine-tuning method with the knowledge distillation algorithm in this paper. Meanwhile, With the Resnet152 model, our method obtained the accuracy of 73.83% on CIFAR100 data and 22x compression, respectively, ResNet110 SVHN achieve 49x compression with 98.23% accuracy and all of which are preferable to the state-of-the-art.