Information-Theoretic Interpretations

Abstract

So far, we have been operating within the theoretical framework of Bayesian inference: the goal of our models is to provide priors for Bayesian inference. An alternative framework is provided by information theory. In information theory, the goal is to find ways of coding the information as efficiently as possible. This turns out to be surprisingly closely connected to Bayesian inference. In many cases, both approaches start by estimation of parameters in a parameterized statistical model.