Determining Principal Component Cardinality Through the Principle of Minimum Description Length

Abstract

PCA (Principal Component Analysis) and its variants are ubiquitous techniques for matrix dimension reduction and reduced-dimension latent-factor extraction. One significant challenge in using PCA, is the choice of the number of principal components. The information-theoretic MDL (Minimum Description Length) principle gives objective compression-based criteria for model selection, but it is difficult to analytically apply its modern definition - NML (Normalized Maximum Likelihood) - to the problem of PCA. This work shows a general reduction of NML problems to lower-dimension problems. Applying this reduction, it bounds the NML of PCA, by terms of the NML of linear regression, which are known.