Optimal weight decay in a perceptron

Abstract

Weight decay was proposed to reduce overfitting as it often appears in the learning tasks of artificial neural networks. In this paper weight decay is applied to a well defined model system based on a single layer perceptron, which exhibits strong overfitting. Since the optimal non-overfitting solution is known for this system, we can compare the effect of the weight decay with this solution. A strategy to find the optimal weight decay strength is proposed, which leads to the optimal solution for any number of examples.