Heuristics of instability and stabilization in model selection

  • Breiman L
  • 154


    Mendeley users who have this article in their library.
  • 545


    Citations of this article.


In model selection, usually a "best" predictor is chosen from a collection μ^(⋅,s){\hat{\mu}(\cdot, s)} of predictors where μ^(⋅,s)\hat{\mu}(\cdot, s) is the minimum least-squares predictor in a collection Us\mathsf{U}_s of predictors. Here s is a complexity parameter; that is, the smaller s, the lower dimensional/smoother the models in Us\mathsf{U}_s. If L\mathsf{L} is the data used to derive the sequence μ^(⋅,s){\hat{\mu}(\cdot, s)}, the procedure is called unstable if a small change in L\mathsf{L} can cause large changes in μ^(⋅,s){\hat{\mu}(\cdot, s)}. With a crystal ball, one could pick the predictor in μ^(⋅,s){\hat{\mu}(\cdot, s)} having minimum prediction error. Without prescience, one uses test sets, cross-validation and so forth. The difference in prediction error between the crystal ball selection and the statistician's choice we call predictive loss. For an unstable procedure the predictive loss is large. This is shown by some analytics in a simple case and by simulation results in a more complex comparison of four different linear regression methods. Unstable procedures can be stabilized by perturbing the data, getting a new predictor sequence μ′^(⋅,s){\hat{\mu'}(\cdot, s)} and then averaging over many such predictor sequences.

Author-supplied keywords

  • Cross-validation
  • Prediction error
  • Predictive loss
  • Regression
  • Subset selection

Get free article suggestions today

Mendeley saves you time finding and organizing research

Sign up here
Already have an account ?Sign in

Find this document


  • Leo Breiman

Cite this document

Choose a citation style from the tabs below

Save time finding and organizing research with Mendeley

Sign up for free