IDEAS home Printed from
   My bibliography  Save this article

Model selection criteria based on cross-validatory concordance statistics


  • Patrick Ten Eyck

    () (The University of Iowa)

  • Joseph E. Cavanaugh

    (The University of Iowa)


Abstract In the logistic regression framework, we present the development and investigation of three model selection criteria based on cross-validatory analogues of the traditional and adjusted c-statistics. These criteria are designed to estimate three corresponding measures of predictive error: the model misspecification prediction error, the fitting sample prediction error, and the sum of prediction errors. We aim to show that these estimators serve as suitable model selection criteria, facilitating the identification of a model that appropriately balances goodness-of-fit and parsimony, while achieving generalizability. We examine the properties of the selection criteria via an extensive simulation study designed as a factorial experiment. We then employ these measures in a practical application based on modeling the occurrence of heart disease.

Suggested Citation

  • Patrick Ten Eyck & Joseph E. Cavanaugh, 2018. "Model selection criteria based on cross-validatory concordance statistics," Computational Statistics, Springer, vol. 33(2), pages 595-621, June.
  • Handle: RePEc:spr:compst:v:33:y:2018:i:2:d:10.1007_s00180-017-0766-7
    DOI: 10.1007/s00180-017-0766-7

    Download full text from publisher

    File URL:
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    1. Wei Pan, 2001. "Akaike's Information Criterion in Generalized Estimating Equations," Biometrics, The International Biometric Society, vol. 57(1), pages 120-125, March.
    2. Mithat Gonen & Glenn Heller, 2005. "Concordance probability and discriminatory power in proportional hazards regression," Biometrika, Biometrika Trust, vol. 92(4), pages 965-970, December.
    3. Hamparsum Bozdogan, 1987. "Model selection and Akaike's Information Criterion (AIC): The general theory and its analytical extensions," Psychometrika, Springer;The Psychometric Society, vol. 52(3), pages 345-370, September.
    4. Makio Ishiguro & Yosiyuki Sakamoto & Genshiro Kitagawa, 1997. "Bootstrapping Log Likelihood and EIC, an Extension of AIC," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 49(3), pages 411-434, September.
    5. Cavanaugh, Joseph E., 1999. "A large-sample model selection criterion based on Kullback's symmetric divergence," Statistics & Probability Letters, Elsevier, vol. 42(4), pages 333-343, May.
    6. Patrick J. Heagerty & Yingye Zheng, 2005. "Survival Model Predictive Accuracy and ROC Curves," Biometrics, The International Biometric Society, vol. 61(1), pages 92-105, March.
    7. Hui Zou & Trevor Hastie, 2005. "Addendum: Regularization and variable selection via the elastic net," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 67(5), pages 768-768, November.
    8. Bengtsson, Thomas & Cavanaugh, Joseph E., 2006. "An improved Akaike information criterion for state-space model selection," Computational Statistics & Data Analysis, Elsevier, vol. 50(10), pages 2635-2654, June.
    9. Hui Zou & Trevor Hastie, 2005. "Regularization and variable selection via the elastic net," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 67(2), pages 301-320, April.
    10. Vieu, Philippe, 1994. "Choice of regressors in nonparametric estimation," Computational Statistics & Data Analysis, Elsevier, vol. 17(5), pages 575-594, June.
    Full references (including those not matched with items on IDEAS)


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:compst:v:33:y:2018:i:2:d:10.1007_s00180-017-0766-7. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Sonal Shukla) or (Mallaigh Nolan). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.