Entropy for semi-Markov processes with Borel state spaces: Asymptotic equirepartition properties and invariance principles

Abstract

The aim of this paper is to define the entropy rate of a semi-Markov process with a Borel state space by extending the strong asymptotic equirepartition property (also called the ergodic theorem of information theory or Shannon-McMillan-Breiman theorem) to this class of non-stationary processes. The mean asymptotic equirepartition property (also called the Shannon-McMillan theorem) is also proven to hold. The relative entropy rate between two semi-Markov processes is defined. All earlier results concerning entropy for semi-Markov processes, jump Markov processes and Markov chains thus appear as special cases. Two invariance principles are established for entropy, one for the central limit theorem and the other for the law of the iterated logarithm. © 2006 ISI/BS.