Introduction to Statistical Methods

Abstract

The principle of maximum likelihood is relatively straightforward. As before, we begin with a sample X = (X 1 , . . . , X n) of random variables chosen according to one of a family of probabilities P θ . In addition, f (x|θ), x = (x 1 , . . . , x n) will be used to denote the density function for the data when θ is the true state of nature. Then, the principle of maximum likelihood yields a choice of the estimato θ as the value for the parameter that makes the observed data most probable. Definition 1. The likelihood function is the density function regarded as a function of θ. L(θ|x) = f (x|θ), θ ∈ Θ. (1) The maximum likelihood estimator (MLE) θ(x) = arg max θ L(θ|x). (2) We will learn that especially for large samples, the maximum likelihood estimators have many desirable properties. However, especially for high dimensional data, the likelihood can have many local maxima. Thus, finding the global maximum can be a major computational challenge. This class of estimators has an important property. I θ(x) is a maximum likelihood estimate for θ, then g θ(x)) is a maximum likelihood estimate for g(θ). For example, if θ is a parameter for the variance an θ is the maximum likelihood estimator, the θ is the maximum likelihood estimator for the standard deviation. This flexibility in estimation criterion seen here is not available in the case of unbiased estimators. Typically, maximizing the score function, ln L(θ|x), the logarithm of the likelihood, will be easier. Having the parameter values be the variable of interest is somewhat unusual, so we will next look at several examples of the likelihood function. 2 Examples Example 2 (Bernoulli trials). If the experiment consists of n Bernoulli trial with success probability p, then L(p|x) = p x1 (1 − p) (1−x1) · · · p xn (1 − p) (1−xn) = p (x1+···+xn) (1 − p) n−(x1+···+xn) . ln L(p|x) = ln p(n i=1 x i) + ln(1 − p)(n − n i=1 x i) = n(¯ x ln p + (1 − ¯ x) ln(1 − p)).