Particle swarm optimization versus gradient based methods in optimizing neural network

Abstract

Neural network modelling has become a great interest for many statisticians to be utilized in various types of data as classification, regression, and time series. It also has been applied in many fields as environment, financial, medical, agriculture and climate change. A lot of parametric methods have been developed to predict time series data such as ARIMA and exponential smoothing. However, requirement of residual assumptions causes limitedness of the models. Time series prediction by using neural network been an interesting study in the forecasting problem. In this model, one of the most interesting discussion is about how to get the optimal weights. Various gradient and non-gradient based methods have been applied in obtaining the network weights. Particle swarm optimization is one non-gradient based algorithm inspired by the behaviour of birds and fish flocks, which move to form certain formations without colliding to get the best position in a multi-dimensional space. In neural network modelling, the number of input and hidden unit give influence to the network architecture. The more complex an architecture, the more network weights must be estimated. In this study, a comparison of particle swarm optimization and some gradient based methods on the optimizing neural network was studied. Comparative studies were performed on both stationary and non-stationary data. Experiments were conducted several times to obtain optimal accuracy and stability of results, through statistics of mean and variance of MSE values.