A penalized approach to mixed model selection via cross-validation

Mixed models play an important role for describing data in various fields, and accordingly selecting the most appropriate mixed model is an appealing topic in model selection literature. To achieve the goal of selecting the most appropriate mixed model, we propose a procedure to jointly select the fixed and random effects by implementing the adaptive Lasso (Zou 2006) penalized methodology via cross-validation. In the procedure, the application of cross-validation can effectively lower the risk of selecting overfitting models. The data are divided into training and test sets, where the training set is utilized for constructing candidate models and the test set is utilized for choosing the most appropriate mixed model. To boost the computational efficiency in the estimation and in the selection of mixed models, we adopt the EM algorithm to optimize the penalized likelihood. Theoretical properties are founded to prove that the proposed approach possesses the consistency and oracle properties. The simulations and a real data example are provided to justify the validity of the procedure.