Non-linear activation function approximation using a REMEZ algorithm

Abstract

Here a more accurate piecewise approximation (PWA) scheme for non-linear activation function is proposed. It utilizes a precision-controlled recursive algorithm to predict a sub-range; after that, the REMEZ algorithm is used to find the corresponding approximation function. The PWA realized in three ways: using first-order functions, that is, piecewise linear model, second-order functions (piecewise non-linear model), and hybrid-order model (a mixture of first-order and second-order functions). The hybrid-order approximation employs the second-order derivative of non-linear activation function to decide the linear and non-linear sub-regions, correspondingly the first-order and second-order functions are predicted, respectively. The accuracy is compared to the present state-of-the-art approximation schemes. The multi-layer perceptron model is designed to implement XOR-gate, and it uses an approximate activation function. The hardware utilization is measured using the TSMC 0.18-μm library with the Synopsys Design Compiler. Result reveals that the proposed approximation scheme efficiently approximates the non-linear activation functions.