Degrees of Freedom: Search Cost and Self-Consistency

Abstract

Model degrees of freedom ((Formula presented.)) is a fundamental concept in statistics because it quantifies the flexibility of a fitting procedure and is indispensable in model selection. To investigate the gap between (Formula presented.) and the number of independent variables in the fitting procedure, Tibshirani introduced the search degrees of freedom ((Formula presented.)) concept to account for the search cost during model selection. However, this definition has two limitations: it does not consider fitting procedures in augmented spaces and does not use the same fitting procedure for (Formula presented.) and (Formula presented.). We propose a modified search degrees of freedom ((Formula presented.)) to directly account for the cost of searching in either original or augmented spaces. We check this definition for various fitting procedures, including classical linear regressions, spline methods, adaptive regressions (the best subset and the lasso), regression trees, and multivariate adaptive regression splines (MARS). In many scenarios when (Formula presented.) is applicable, (Formula presented.) reduces to (Formula presented.). However, for certain procedures like the lasso, (Formula presented.) offers a fresh perspective on search costs. For some complex procedures like MARS, the (Formula presented.) has been pre-determined during model fitting, but the (Formula presented.) of the final fitted procedure might differ from the pre-determined one. To investigate this discrepancy, we introduce the concepts of nominal (Formula presented.) and actual (Formula presented.), and define the property of self-consistency, which occurs when there is no gap between these two (Formula presented.) ’s. We propose a correction procedure for MARS to align these two (Formula presented.) ’s, demonstrating improved fitting performance through extensive simulations and two real data applications. Supplementary materials for this article are available online.