Applying mutual information to adaptive mixture models

Abstract

This paper presents a method for determine an optimal set of components for a density mixture model using mutual information. A component with small mutual information is believed to be independent from the rest components and to make a significant contribution to the system and hence cannot be removed. Whilst a component with large mutual information is believed to be unlikely independent from the rest components within a system and hence can be removed. Continuing removing components with positive mutual information till the system mutual information becomes non-positive will finally give rise to a parsimonious structure for a density mixture model. The method has been verified with several examples.