Model selection and weight sharing of multi-layer perceptrons

Abstract

We present a method to learn and select a succinct multi-layer perceptron having shared weights. Weight sharing means a weight is allowed to have one of common weights. A near-zero common weight can be eliminated, called weight pruning. Our method iteratively merges and splits common weights based on 2nd-order criteria, escaping local optima through bidirectional clustering. Moreover, our method selects the optimal number of hidden units based on cross-validation. Our experiments showed that the proposed method can perfectly restore the original sharing structure for an artificial data set, and finds a small number of common weights for a real data set. © Springer-Verlag Berlin Heidelberg 2005.