Data-Aware Customization of Activation Functions Reduces Neural Network Error

Abstract

Activation functions play a critical role in neural networks, yet off-the-shelf models often pay little attention to their specific choice. Here, we show that data-aware customization of activation functions can lead to striking reductions in neural network error. We begin with a simple linear algebraic explanation of the role of activation functions, and, through a connection to the Diaconis-Shahshahani Approximation Theorem, propose a set of criteria for effective activation functions. As a case study, we consider regression tasks with a partially exchangeable target function, i.e., f(u,v,w)=f(v,u,w) for u,v∈Rd, and w∈Rk, and prove that using an even activation function in at least one layer ensures the prediction preserves partial exchangeability for improved performance. Since even activation functions are rarely used in practice, we design a novel “seagull” activation function, log(1+x2), guided by our criteria. Empirical tests on over two dozen regression tasks with varying local smoothness, curvature, and degrees of exchangeability demonstrate that substituting the seagull function, even in a well-trained model, can reduce error by an order of magnitude. This improvement is most pronounced when the substitution occurs in the layer where exchangeable variables are first combined. While especially beneficial in low-dimensional data, tests on CIFAR-10 show noticeable gains even in high-dimensional settings. These results underscore the general potential of customizing activation functions to improve neural network performance.