Performance analysis of various activation functions in generalized MLP architectures of neural networks

Abstract

The activation function used to transform the activati on level of a unit (neuron) into an output signal. There are a number of common activation functi ons in use with artificial neural networks (ANN). The most common choice of activation functions fo r multi layered perceptron (MLP) is used as transfer functions in research and engineerin g. Among the reasons for this popularity are its boundedness in the unit interval, the function’s and its derivative’s fast computability, and a number of amenable mathematical properties in the re alm of approximation theory. However, considering the huge variety of problem domains MLP i s applied in, it is intriguing to suspect that specific problems call for single or a set of specific a ctivation functions. The aim of this study is to analyze the performance of generalized MLP architectures which has back-propagation algorithm using various different activation functions for the neurons of hidden and output layers. For experimental comparisons, Bi-polar sigmoid, Uni-pola r sigmoid, Tanh, Conic Section, and Radial Bases Function (RBF) were used.