Independent Components in Text

Abstract

In this communication we analyze the feasibility ofindependent component analysis (ICA) for dimensionalreduction and representation of word histograms. Theanalysis is carried out in a likelihood frameworkwhich allows estimates of the loadings (sourcesignals), the mixing matrix and the noise level. Inthe face of noisy signals, the estimated sources arenon-linear functionals of the observed signals, incontrast to the linear noise free case. We alsodiscuss the generalizability of the estimated modelsand show that an empirical test error estimate maybe used to optimize model dimensionality, inparticular the optimal number of sources. Whenapplied to word histograms ICA is shown to producerepresentations that are better aligned with thegroup structure in the text data than the LSA.