A Study on Different Functionalities and Performances among Different Activation Functions across Different ANNs for Image Classification

Abstract

Activation functions are very essential in artificial neural networks (ANNs), since they are non-linear functions and they have been proved necessary to implement deep learning. Recently, ReLU is one of the most well-known activation functions; however, several competitors - e.g. LReLU and SWISH - have nowadays been proposed or 'discovered'. In this paper, the authors perform a detailed comparison of five activation functions over two image classification datasets. We found the overall performances of accuracy rates ranked from the best GELU, RELU, SWISH, SELU, down to Sigmoid. Such the observation would result in the improvement of future image classification via designing new state-of-the-art activation functions.