Relative Entropy and Csiszar’s Projection

Abstract

This chapter is devoted to a brief exposition of relative entropy, and to proofs of Csiszar's projection theorem and those on exponential families needed in Chapter 5. 10.1. Relative Entropy Let {n, B} be a measurable space, MI (n) : the space of probability measures on n, and the relative entropyl H(QIP) of Q with respect to P be defined by (10.1) H(QIP)=fClOgdQ)dQ, ifQ«P, (=00, otherwise). dP where P, Q E MI(n). Denoting the Radon-Nikodym derivative of Q with respect to P as q = qp = if, , the relative entropy H(Q I P) defined by (10.1) can be written in terms of the density function q as 1 It has appeared under various names, cf. Kullback (1959), Csiszar (1975,84) M. Nagasawa, Schrödinger Equations and Diffusion Theory