An exploration of the power of max switch locations in CNNs

Abstract

In this paper, we present the power of max switch locations in convolutional neural networks (CNNs) with two experiments: image reconstruction and classification. First, we realize image reconstruction via a convolutional auto-encoder (CAE) that includes max pooling/unpooling operations in an encoder and decoder, respectively. During decoder operation, we alternate max switch locations extracted from another image, which was chosen from among the real images and noise images. Meanwhile, we set up a classification experiment in a teacher-student manner, allowing the transmission of max switch locations from the teacher network to the student network. During both the training and test phases, we let the student network receive max switch locations from the teacher network, and we observe prediction similarity for both networks while the input to the student network is either randomly shuffled test data or a noise image. Based on the results of both experiments, we conjecture that max switch locations could be another form of distilled knowledge. In a teacher-student scheme, therefore, we present a new max pool method whereby the distilled knowledge improves the performance of the student network in terms of training speed. We plan to implement this method in future work.