Response surface of neural networks learned using Bayesian inference and its application to optimization problem

Abstract

This paper verifies the response surfaces of artificial neural networks (NN) learned by using a method based on Bayesian inference. Mackay showed that the Bayesian method due to Gall and Skilling can be applied to regularization for NN. However, generalization ability has not been verified sufficiently for the NN response surface regularized by using the Bayesian method. If the NN response surface has good generalization ability, it can be used in the optimization process of response surface methodology (RSM). NN therefore was learned by using the Bayesian method to investigate generalization ability. We tried three rules for updating the regularizing constants in an objective function minimized during NN learning. All of the update rules were derived from the Bayesian method. As a result, the response surface of NN had good generalization ability, with the exception of one update rule. The poor update rule failed to determine the regularizing constants. This tendency for the update rules was recognized regardless of response surface geometry. After we selected a superior update rule, the NN response surface by using the Bayesian method was applied to an optimization problem. The response surface didn't fit noises included in teacher data, and consequently, it was effectively used to reach a solution. Finally, we concluded that the NN learned by using the Bayesian method can be used as the response surface in the process of RSM.