K-Fold Cross-Validation is Superior to Split Sample Validation for Risk Adjustment Models

This paper examines cross-validation techniques, with a particular focus on assessing thepredictive validity of risk adjustment models as commonly estimated. We validate that K-Fold cross-validation is more efficient than a 50-50 split sample and illustrate that overfitting with rich risk adjustment models remains meaningful even in samples of a million observations. A new estimation algorithm is described that efficiently calculates K-Fold cross-validated R-squared and other measures of goodness of fit using only three (XXX verify) passes through the data, and hence can be applied easily on sample sizes in the millions without sorting or relying on repeated split-sample techniques. Analysis of K-fold cross-validation results using a large claims dataset is used to calculate the standard deviation and bias of fitted R-squares for different models and sample sizes, which have a larger bias in moderately large sample sizes than most researchers would realize. Programs that implement the algorithm in SAS and STATA are presented that can be easily used on any sample.