Benchmarking Comparison of Swish vs. Other Activation Functions on CIFAR-10 Imageset

Abstract

The choice of the most appropriate activation functions for artificial neural networks has a significant effect on the training time and task performance. Nowadays the most widely-used activation function is the Rectified Linear Unit (ReLU). Despite its “dying ReLU problem” and many attempts to replace it with something better, it is still considered as default choice to begin with creation of network. Two years ago a new, promising function has been described formulated by Google Brain Team. The proposed function - named Swish - was obtained using a combination of exhaustive and reinforcement learning-based search. According to the authors, simply replacing ReLUs with Swish units improves top-1 classification accuracy on ImageNet by 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2. This paper describes an experiment on CIFAR-10 image set where Swish appears not to outperform ReLU.