Complexity of shallow networks representing finite mappings

Abstract

Complexity of shallow (one-hidden-layer) networks representing finitemultivariate mappings is investigated. Lower bounds are derived on growth of numbers of network units and sizes of output weights in terms of variational norms of mappings to be represented. Probability distributions of mappings whose computations require large networks are described. It is shown that due to geometrical properties of highdimensional Euclidean spaces, representation of almost any randomly chosen function on a sufficiently large domain by a shallow network with perceptrons requires untractably large network. Concrete examples of such functions are constructed using Hadamard matrices.