Context sensitive information: Model validation by information theory

Abstract

A theory of patterns analysis has to provide a criterion to filter out the relevant information to identify patterns. The set of potential patterns, also called hypothesis class of the problem, defines admissible explanations of the available data and it specifies the context for a patterns analysis task. Fluctuations in the measurements limit the precision which we can achieve to identify such patterns. Effectively, the distinguishible patterns define a code in a fictitious communication scenario where the selected cost function together with a stochastic data source plays the role of a noisy "channel". Maximizing the capacity of this channel determines the penalized costs of the pattern analysis problem with a data dependent regularization strength. The tradeoff between informativeness and robustness in statistical inference is mirrored in the balance between high information rate and zero communication error, thereby giving rise to a new notion of context sensitive information. © 2011 Springer-Verlag Berlin Heidelberg.