Parametric Activation Functions for Neural Networks: A Tutorial Survey

Abstract

Activation functions are pivotal in neural networks, determining the output of each neuron. Traditionally, functions like sigmoid and ReLU have been static and deterministic. However, the introduction of parametric activation functions represents a significant paradigm shift by incorporating adjustable parameters, thus enhancing flexibility and optimization capabilities. Furthermore, parametric activation functions can enhance interpretability, helping to solve the black-box problem and providing deeper insights into the functioning of neural networks. They are also essential components of the innovative model of Kolmogorov-Arnold Network, that features learnable activation functions on the edges of the neural network. This review highlights the importance, characteristics, and applications of these parametric functions in neural networks. Notable examples of parametric activation functions are analyzed, with emphasis on their mathematical formulations and their transformative impact on modern neural networks. These functions not only represent an evolutionary step but a substantial shift in the design and optimization of neural network architectures, offering a promising avenue for innovation and advancement in the field.