Phase Transitions in Rate Distortion Theory and Deep Learning

Abstract

Rate distortion theory is concerned with optimally encoding signals from a given signal class S using a budget of R bits, as R→ ∞. We say that Scan be compressed at rates if we can achieve an error of at most O(R-s) for encoding the given signal class; the supremal compression rate is denoted by s∗(S). Given a fixed coding scheme, there usually are some elements of S that are compressed at a higher rate than s∗(S) by the given coding scheme; in this paper, we study the size of this set of signals. We show that for certain “nice” signal classes S, a phase transition occurs: We construct a probability measure P on S such that for every coding scheme C and any s> s∗(S) , the set of signals encoded with error O(R-s) by C forms a P-null-set. In particular, our results apply to all unit balls in Besov and Sobolev spaces that embed compactly into L2(Ω) for a bounded Lipschitz domain Ω. As an application, we show that several existing sharpness results concerning function approximation using deep neural networks are in fact generically sharp. In addition, we provide quantitative and non-asymptotic bounds on the probability that a random f∈ S can be encoded to within accuracy ε using R bits. This result is subsequently applied to the problem of approximately representing f∈ S to within accuracy ε by a (quantized) neural network with at most W nonzero weights. We show that for any s> s∗(S) there are constants c, C such that, no matter what kind of “learning” procedure is used to produce such a network, the probability of success is bounded from above by min{1,2C·W⌈log2(1+W)⌉2-c·ε-1/s}.