Model order estimation

Abstract

In the statistical approach known as model selection, a collection of models is given, and the statistical procedure for model selection stems from a compromise between model complexity and adequacy to the data. Different types of adequacy to data can be used, according to the question one is interested in. In some applications, the model order is a structural parameter with a precise interpretation regarding the phenomenon being studied. Here, we will be interested in the statistical estimation of a model’s order, as well as its link with universal coding. We start by recalling the general framework of model selection, and briefly describe the MDL principle, as introduced by Rissanen, according to which “sparse coding” leads to a good compromise between complexity and adequacy. We will then consider more specifically the question of inferring the order of a model, focusing on two types of model collections: hidden Markov chains models, and population mixture models. We study in detail penalized maximum likelihood estimators. As it will be revealed, we will need to understand the likelihood’s fluctuations, and we will see why this is a hard question for hidden Markov chains models and for population mixture models. We will see how results in universal coding allow for a first analysis of the likelihood’s fluctuations and help to obtain consistency results, in particular for hidden Markov chains. However, this analysis is still sub-optimal, and the end of the chapter will be devoted to the study of independent random variables, in particular in population mixture models. In this situation, it is possible to carry out a precise study of the likelihood’s fluctuations.