IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0221433.html
   My bibliography  Save this article

Cumulative ROC curves for discriminating three or more ordinal outcomes with cutpoints on a shared continuous measurement scale

Author

Listed:
  • B Rey deCastro

Abstract

Cumulative receiver operator characteristic (ROC) curve analysis extends classic ROC curve analysis to discriminate three or more ordinal outcome levels on a shared continuous scale. The procedure combines cumulative logit regression with a cumulative extension to the ROC curve and performs as expected with ternary (three-level) ordinal outcomes under a variety of simulated conditions (unbalanced data, proportional and non-proportional odds, areas under the ROC curve [AUCs] from 0.70 to 0.95). Simulations also compared several criteria for selecting cutpoints to discriminate outcome levels: the Youden Index, Matthews Correlation Coefficient, Total Accuracy, and Markedness. Total Accuracy demonstrated the least absolute percent-bias. Cutpoints computed from maximum likelihood regression parameters demonstrated bias that was often negligible. The procedure was also applied to publicly available data related to computer imaging and biomarker exposure science, yielding good to excellent AUCs, as well as cutpoints with sensitivities and specificities of commensurate quality. Implementation of cumulative ROC curve analysis and extension to more than three outcome levels are straightforward. The author’s programs for ternary ordinal outcomes are publicly available.

Suggested Citation

  • B Rey deCastro, 2019. "Cumulative ROC curves for discriminating three or more ordinal outcomes with cutpoints on a shared continuous measurement scale," PLOS ONE, Public Library of Science, vol. 14(8), pages 1-16, August.
    • Handle: RePEc:plo:pone00:0221433
    DOI: 10.1371/journal.pone.0221433
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0221433
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0221433&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0221433?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Anna N. Angelos Tosteson & Colin B. Begg, 1988. "A General Regression Methodology for ROC Curve Estimation," Medical Decision Making, , vol. 8(3), pages 204-215, August.
    2. Hirschberg, Joe & Lye, Jenny, 2010. "A Geometric Comparison of the Delta and Fieller Confidence Intervals," The American Statistician, American Statistical Association, vol. 64(3), pages 234-241.
    3. Jing Qin, 2003. "Using logistic regression procedures for estimating receiver operating characteristic curves," Biometrika, Biometrika Trust, vol. 90(3), pages 585-596, September.
    4. Jan Budczies & Frederick Klauschen & Bruno V Sinn & Balázs Győrffy & Wolfgang D Schmitt & Silvia Darb-Esfahani & Carsten Denkert, 2012. "Cutoff Finder: A Comprehensive and Straightforward Web Application Enabling Rapid Biomarker Cutoff Optimization," PLOS ONE, Public Library of Science, vol. 7(12), pages 1-7, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Joe Hirschberg & Jenny Lye, 2017. "Alternative Graphical Representations of the Confidence Intervals for the Structural Coefficient from Exactly Identified Two-Stage Least Squares," Department of Economics - Working Papers Series 2026, The University of Melbourne.
    2. Jin, Hua & Lu, Ying, 2009. "The ROC region of a regression tree," Statistics & Probability Letters, Elsevier, vol. 79(7), pages 936-942, April.
    3. Ziyi Li & Yijian Huang & Dattatraya Patil & Martin G. Sanda, 2023. "Covariate adjustment in continuous biomarker assessment," Biometrics, The International Biometric Society, vol. 79(1), pages 39-48, March.
    4. Carson, Richard T. & Czajkowski, Mikołaj, 2019. "A new baseline model for estimating willingness to pay from discrete choice models," Journal of Environmental Economics and Management, Elsevier, vol. 95(C), pages 57-61.
    5. Jin Sheng & Yun-Peng Yang & Yu-Xiang Ma & Tao Qin & Zhi-Huang Hu & Shao-Dong Hong & Ting Zhou & Yan Huang & Hong-Yun Zhao & Li Zhang, 2016. "Low Prognostic Nutritional Index Correlates with Worse Survival in Patients with Advanced NSCLC following EGFR-TKIs," PLOS ONE, Public Library of Science, vol. 11(1), pages 1-12, January.
    6. Wilbert B. van den Hout, 2003. "The Area under an ROC Curve with Limited Information," Medical Decision Making, , vol. 23(2), pages 160-166, March.
    7. repec:jss:jstsof:08:i12 is not listed on IDEAS
    8. Hirschberg, Joe & Lye, Jenny, 2017. "Inverting the indirect—The ellipse and the boomerang: Visualizing the confidence intervals of the structural coefficient from two-stage least squares," Journal of Econometrics, Elsevier, vol. 199(2), pages 173-183.
    9. William M. Briggs & Russell Zaretzki, 2008. "The Skill Plot: A Graphical Technique for Evaluating Continuous Diagnostic Tests," Biometrics, The International Biometric Society, vol. 64(1), pages 250-256, March.
    10. Rodríguez-Álvarez, María Xosé & Roca-Pardiñas, Javier & Cadarso-Suárez, Carmen, 2011. "A new flexible direct ROC regression model: Application to the detection of cardiovascular risk factors by anthropometric measures," Computational Statistics & Data Analysis, Elsevier, vol. 55(12), pages 3257-3270, December.
    11. Luo, Jingqin & Xiong, Chengjie, 2012. "DiagTest3Grp: An R Package for Analyzing Diagnostic Tests with Three Ordinal Groups," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 51(i03).
    12. Cheam, Amay S.M. & McNicholas, Paul D., 2016. "Modelling receiver operating characteristic curves using Gaussian mixtures," Computational Statistics & Data Analysis, Elsevier, vol. 93(C), pages 192-208.
    13. Nasim Vahabi & Anoshirvan Kazemnejad & Somnath Datta, 2018. "A Marginalized Overdispersed Location Scale Model for Clustered Ordinal Data," Sankhya B: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 80(1), pages 103-134, December.
    14. Zhang, Biao, 2006. "A semiparametric hypothesis testing procedure for the ROC curve area under a density ratio model," Computational Statistics & Data Analysis, Elsevier, vol. 50(7), pages 1855-1876, April.
    15. Jean-Marie Dufour & Emmanuel Flachaire & Lynda Khalaf & Abdallah Zalghout, 2020. "Identification-Robust Inequality Analysis," Cahiers de recherche 03-2020, Centre interuniversitaire de recherche en économie quantitative, CIREQ.
    16. Claudia Bühnemann & Simon Li & Haiyue Yu & Harriet Branford White & Karl L Schäfer & Antonio Llombart-Bosch & Isidro Machado & Piero Picci & Pancras C W Hogendoorn & Nicholas A Athanasou & J Alison No, 2014. "Quantification of the Heterogeneity of Prognostic Cellular Biomarkers in Ewing Sarcoma Using Automated Image and Random Survival Forest Analysis," PLOS ONE, Public Library of Science, vol. 9(9), pages 1-14, September.
    17. Hyungjin Kim & Chang Min Park & Bhumsuk Keam & Sang Joon Park & Miso Kim & Tae Min Kim & Dong-Wan Kim & Dae Seog Heo & Jin Mo Goo, 2017. "The prognostic value of CT radiomic features for patients with pulmonary adenocarcinoma treated with EGFR tyrosine kinase inhibitors," PLOS ONE, Public Library of Science, vol. 12(11), pages 1-13, November.
    18. Jiang, Shan & Tu, Dongsheng, 2012. "Inference on the probability P(T1," Computational Statistics & Data Analysis, Elsevier, vol. 56(5), pages 1069-1078.
    19. Joe Hirschberg and Jenny Lye, 2009. "Notes on the Construction of Geometric Representations of Confidence Intervals of Ratios using Stata, Gauss and Eviews," Department of Economics - Working Papers Series 1061, The University of Melbourne.
    20. Margaret Sullivan Pepe, 2008. "Discussions," Biometrics, The International Biometric Society, vol. 64(1), pages 256-258, March.
    21. Alicja Jokiel-Rokita & Rafał Topolnicki, 2019. "Minimum distance estimation of the binormal ROC curve," Statistical Papers, Springer, vol. 60(6), pages 2161-2183, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0221433. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.