The Strong Ergodic Theorem for Densities: Generalized Shannon-McMillan-Breiman Theorem

Abstract

Let { X 1 , X 2 , ⋯ } be a stationary process with probability densities f ( X 1 , X 2 , ⋯ , X n ) with respect to Lebesgue measure or with respect to a Markov measure with a stationary transition measure. It is shown that the sequence of relative entropy densities ( 1 / n ) log f ( X 1 , X 2 , ⋯ , X n ) converges almost surely. This long-conjectured result extends the L 1 convergence obtained by Moy, Perez, and Kieffer and generalizes the Shannon-McMillan-Breiman theorem to nondiscrete processes. The heart of the proof is a new martingale inequality which shows that logarithms of densities are L 1 dominated.