The Exponential Family and Statistical Applications

Abstract

The exponential family is a practically convenient and widely used unified family of distributions on finite-dimensional Euclidean spaces parametrized by a finite-dimensional parameter vector. Specialized to the case of the real line, the exponential family contains as special cases most of the standard discrete and continuous distributions that we use for practical modeling, such as the normal, Poisson, binomial, exponential, Gamma, multivariate normal, and so on. The reason for the special status of the exponential family is that a number of important and useful calculations in statistics can be done all at one stroke within the framework of the exponential family. This generality contributes to both convenience and larger-scale understanding. The exponential family is the usual testing ground for the large spectrum of results in parametric statistical theory that require notions of regularity or Cramér–Rao regularity. In addition, the unified calculations in the exponential family have an element of mathematical neatness. Distributions in the exponential family have been used in classical statistics for decades. However, it has recently obtained additional importance due to its use and appeal to the machine learning community. A fundamental treatment of the general exponential family is provided in this chapter. Classic expositions are available in Barndorff-Nielsen (1978), Brown (1986), and Lehmann and Casella (1998). An excellent recent treatment is available in Bickel and Doksum (2006).