IDEAS home Printed from https://ideas.repec.org/a/eee/csdana/v56y2012i6p1966-1980.html

Assessing the predictive ability of a multilevel binary regression model

Author

Listed:
  • Van Oirbeek, R.
  • Lesaffre, E.

Abstract

An adaptation of the Brier score and the concordance probability is proposed for the two-level and the three-level random intercept binary regression model. This results in 2 different Brier scores and 3 different C-indices for the two-level binary regression model and 4 different Brier scores and 7 different C-indices for the three-level binary regression model. The ensemble of these measures offers a better view on how the different elements of the random effects model, i.e. the covariates and the random effects, affect the predictive ability of the model separately, evaluated on a within-cluster, between-cluster and global level. For all measures, an estimation procedure using Bayesian and likelihood estimation methods was developed, including a percentile and a BCa non-parametric bootstrap step to construct credible/confidence intervals. In a simulation study, the likelihood estimation procedure showed difficulties in estimating unbiasedly the predictive ability of the random effects, while the Bayesian estimation procedure resulted in good estimation properties for all of the developed measures. The BCa non-parametric bootstrap method resulted in confidence/credible intervals with better coverage properties than the percentile non-parametric bootstrap method. The proposals are applied to a real-life binary data set with a three-level structure using the Bayesian estimation procedure.

Suggested Citation

  • Van Oirbeek, R. & Lesaffre, E., 2012. "Assessing the predictive ability of a multilevel binary regression model," Computational Statistics & Data Analysis, Elsevier, vol. 56(6), pages 1966-1980.
    • Handle: RePEc:eee:csdana:v:56:y:2012:i:6:p:1966-1980
    DOI: 10.1016/j.csda.2011.11.023
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167947311004221
    Download Restriction: Full text for ScienceDirect subscribers only.
    File URL: https://libkey.io/10.1016/j.csda.2011.11.023?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Germán Rodríguez & Noreen Goldman, 2001. "Improved estimation procedures for multilevel models with binary response: a case‐study," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 164(2), pages 339-355.
    2. Thomas A. Gerds & Martin Schumacher, 2007. "Efron-Type Measures of Prediction Error for Survival Analysis," Biometrics, The International Biometric Society, vol. 63(4), pages 1283-1287, December.
    3. C. A. Field & A. H. Welsh, 2007. "Bootstrapping clustered data," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 69(3), pages 369-390, June.
    4. Anders Skrondal & Sophia Rabe‐Hesketh, 2009. "Prediction in multilevel generalized linear models," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 172(3), pages 659-687, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Robin Van Oirbeek & Emmanuel Lesaffre, 2018. "An Investigation of the Discriminatory Ability of the Clustering Effect of the Frailty Survival Model," Biostatistics and Biometrics Open Access Journal, Juniper Publishers Inc., vol. 6(3), pages 87-98, April.
    2. Veronica Distefano & Sabrina Maggio & Sandra De Iaco, 2025. "Multilevel Modeling of Training Needs in Artificial Intelligence," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 178(3), pages 1411-1439, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Robin Van Oirbeek & Emmanuel Lesaffre, 2018. "An Investigation of the Discriminatory Ability of the Clustering Effect of the Frailty Survival Model," Biostatistics and Biometrics Open Access Journal, Juniper Publishers Inc., vol. 6(3), pages 87-98, April.
    2. Emilia Justyna Powell & Steven Christian McDowell & Robert O’Brien & Julia Oksasoglu, 2021. "Islam-based legal language and state governance: democracy, strength of the judiciary and human rights," Constitutional Political Economy, Springer, vol. 32(3), pages 376-412, September.
    3. Weible, Daniela & Salamon, Petra & Christoph-Schulz, Inken B. & Peter, Guenter, 2013. "How do political, individual and contextual factors affect school milk demand? Empirical evidence from primary schools in Germany," Food Policy, Elsevier, vol. 43(C), pages 148-158.
    4. Wen Shi & Xi Chen & Jennifer Shang, 2019. "An Efficient Morris Method-Based Framework for Simulation Factor Screening," INFORMS Journal on Computing, INFORMS, vol. 31(4), pages 745-770, October.
    5. David Roodman & James G. MacKinnon & Morten Ørregaard Nielsen & Matthew D. Webb, 2019. "Fast and wild: Bootstrap inference in Stata using boottest," Stata Journal, StataCorp LLC, vol. 19(1), pages 4-60, March.
    6. Nils Gutacker & Andrew Street, 2018. "Multidimensional performance assessment of public sector organisations using dominance criteria," Health Economics, John Wiley & Sons, Ltd., vol. 27(2), pages 13-27, February.
    7. Sun-Joo Cho & Paul Boeck & Susan Embretson & Sophia Rabe-Hesketh, 2014. "Additive Multilevel Item Structure Models with Random Residuals: Item Modeling for Explanation and Item Generation," Psychometrika, Springer;The Psychometric Society, vol. 79(1), pages 84-104, January.
    8. Schurer, Stefanie & Yong, Jongsay, 2012. "Personality, well-being and the marginal utility of income: What can we learn from random coefficient models?," Working Paper Series 18617, Victoria University of Wellington, School of Economics and Finance.
    9. Steffen Nestler & Sarah Humberg, 2022. "A Lasso and a Regression Tree Mixed-Effect Model with Random Effects for the Level, the Residual Variance, and the Autocorrelation," Psychometrika, Springer;The Psychometric Society, vol. 87(2), pages 506-532, June.
    10. Coisnon, Thomas & Rousselière, Damien & Rousselière, Samira, 2018. "Information on biodiversity and environmental behaviors: a European study of individual and institutional drivers to adopt sustainable gardening practices," Working Papers 272611, Institut National de la recherche Agronomique (INRA), Departement Sciences Sociales, Agriculture et Alimentation, Espace et Environnement (SAE2).
    11. Janice L. Scealy & David Heslop & Jia Liu & Andrew T. A. Wood, 2022. "Directions Old and New: Palaeomagnetism and Fisher (1953) Meet Modern Statistics," International Statistical Review, International Statistical Institute, vol. 90(2), pages 237-258, August.
    12. Mathieu Bunel & Yannick L'Horty, 2011. "Les effets des aides publiques aux Hôtels Cafés Restaurants et leurs interactions," Working Papers halshs-00658460, HAL.
    13. Matthias Schmid & Thomas Hielscher & Thomas Augustin & Olaf Gefeller, 2011. "A Robust Alternative to the Schemper–Henderson Estimator of Prediction Error," Biometrics, The International Biometric Society, vol. 67(2), pages 524-535, June.
    14. Getinet A. Haile, 2015. "Workplace Job Satisfaction in Britain: Evidence from Linked Employer–Employee Data," LABOUR, CEIS, vol. 29(3), pages 225-242, September.
    15. Orth, Walter, 2012. "The predictive accuracy of credit ratings: Measurement and statistical inference," International Journal of Forecasting, Elsevier, vol. 28(1), pages 288-296.
    16. Gerard J Van Den Berg & Barbara Hofmann & Arne Uhlendorff, 2016. "The Role of Sickness in the Evaluation of Job Search Assistance and Sanctions," Working Papers 2016-17, Center for Research in Economics and Statistics.
    17. Seonho Shin, 2021. "Were they a shock or an opportunity?: The heterogeneous impacts of the 9/11 attacks on refugees as job seekers—a nonlinear multi-level approach," Empirical Economics, Springer, vol. 61(5), pages 2827-2864, November.
    18. Mary Ying-Fang Wang & Paul Tuss & Lihong Qi, 2019. "Augmented Weighted Estimators Dealing with Practical Positivity Violation to Causal inferences in a Random Coefficient Model," Psychometrika, Springer;The Psychometric Society, vol. 84(2), pages 447-467, June.
    19. Elbers, Chris & van der Weide, Roy, 2014. "Estimation of normal mixtures in a nested error model with an application to small area estimation of poverty and inequality," Policy Research Working Paper Series 6962, The World Bank.
    20. Matthias Weber & Martin Schumacher & Harald Binder, 2014. "Regularized Regression Incorporating Network Information: Simultaneous Estimation of Covariate Coefficients and Connection Signs," Tinbergen Institute Discussion Papers 14-089/I, Tinbergen Institute.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:csdana:v:56:y:2012:i:6:p:1966-1980. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/csda .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.