NMS-KSD: Efficient Knowledge Distillation for Dense Object Detection via Non-Maximum Suppression and Feature Storage

Abstract

Recently, many studies have proposed knowledge distillation (KD) frameworks for object detection. However, these frameworks did not take into account the inefficiencies caused by the teacher detector. The inefficiency refers to the computational cost incurred during the process of passing input data to the teacher model to acquire its knowledge. To solve this inefficiency in image classification, Fast Knowledge Distillation (FKD) was proposed, which stores the teacher model’s knowledge in advance and then uses it in the distillation process. However, directly applying FKD’s knowledge storage mechanism to dense object detectors causes a storage space problem. To address this issue, we propose NMS-KSD, a novel knowledge storage distillation method designed for dense object detection tasks. The core of NMS-KSD is the integration of Non-Maximum Suppression (NMS) and channel max pooling to effectively select and store key features from the teacher model’s intermediate feature maps. By storing and reusing these key features, NMS-KSD addresses the inefficiencies of traditional KD frameworks, significantly reducing training time while maintaining high performance. We validate the effectiveness and efficiency of our method across various dense object detectors through extensive experiments on the COCO, PASCAL VOC, and Cityscapes datasets.