Stochastic drop of kernel windows for improved generalization in convolution neural networks

Abstract

We propose a novel dropout technique for convolutional neural networks by redesigned Dropout and DropConnect methods. Conventional drop methods work on the individual single weight value of the fully connected network. When they are applied to convolution layers, only some kernel weights are removed. However, all the weights of the convolutional kernel windows together constitute a specific pattern, so dropping part of kernel window weights may cause change of the learned patterns and may model completely different local patterns. We assign the basic unit of drop method for convolutional weights to be the whole kernel windows, so one output map value is dropped. We evaluated the proposed DropKernel strategy by the object classification performance on CIFAR10 in comparison to conventional Dropout and DropConnect methods, and showed improved performance of the proposed method.