Best-subset model selection based on multitudinal assessments of likelihood improvements

A common model selection approach is to select the best model, according to some criterion, from among the collection of models defined by all possible subsets of the explanatory variables. Identifying an optimal subset has proven to be a challenging problem, both statistically and computationally. Our model selection procedure allows the researcher to nominate, a priori, the probability at which models containing false or spurious variables will be selected from among all possible subsets. The procedure determines whether inclusion of each candidate variable results in a sufficiently improved fitting term – and is hence named the SIFT procedure. Two variants are proposed: a naive method based on a set of restrictive assumptions and an empirical permutation-based method. Properties of these methods are investigated within the standard linear modeling framework and performance is evaluated against other model selection techniques. The SIFT procedure behaves as designed – asymptotically selecting variables that characterize the underlying data generating mechanism, while limiting selection of spurious variables to the desired level. The SIFT methodology offers researchers a promising new approach to model selection, providing the ability to control the probability of selecting a model that includes spurious variables to a level based on the context of the application.