Statistical model choice

Variable selection methods and model selection approaches are valuable statistical tools, which are indispensable for almost any statistical modeling question. This review considers first the use of information criteria for model selection. Such criteria provide an ordering of the considered models where the `best' model is selected. Different modeling goals, though, might require different criteria to be used. Next, the effect of including a penalty in the estimation process is discussed. Also nonparametric estimation contains several aspects of model choice, such as the choice of the estimator to use and the selection of tuning parameters. Fourth, model averaging approaches are reviewed where estimators from different models are weighted to provide one final estimator. There are several ways to choose the weights, most of them result in data-driven, hence random, weights. Challenges for inference after model selection and inference for model averaged estimators are discussed.