A Comparison of GCV and GML for Choosing the Smoothing Parameter in the Generalized Spline Smoothing Problem

Abstract

The partially improper prior behind the smoothing spline model is used to obtain a generalization of the maximum likelihood {(GML)} estimate for the smoothing parameter. Then this estimate is compared with the generalized cross validation {(GCV)} estimate both analytically and by Monte Carlo methods. The comparison is based on a predictive mean square error criteria. It is shown that if the true, unknown function being estimated is smooth in a sense to be defined then the {GML} estimate undersmooths relative to the {GCV} estimate and the predictive mean square error using the {GML} estimate goes to zero at a slower rate than the mean square error using the {GCV} estimate. If the true function is "rough" then the {GCV} and {GML} estimates have asymptotically similar behavior. A Monte Carlo experiment was designed to see if the asymptotic results in the smooth case were evident in small sample sizes. Mixed results were obtained for $n = 32$, {GCV} was somewhat better than {GML} for $n = 64$, and {GCV} was decidedly superior for $n = 128$. In the $n = 32$ case {GCV} was better for smaller \${\textbackslash}sigma{\textasciicircum}2$ and the comparison close for larger \${\textbackslash}sigma{\textasciicircum}2$. The theoretical results are shown to extend to the generalized spline smoothing model, which includes the estimate of functions given noisy values of various integrals of them.