IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0288233.html

External evaluation of the Dynamic Criticality Index: A machine learning model to predict future need for ICU care in hospitalized pediatric patients

Author

Listed:
  • Anita K Patel
  • Eduardo Trujillo-Rivera
  • James M Chamberlain
  • Hiroki Morizono
  • Murray M Pollack

Abstract

Objective: To assess the single site performance of the Dynamic Criticality Index (CI-D) models developed from a multi-institutional database to predict future care. Secondarily, to assess future care-location predictions in a single institution when CI-D models are re-developed using single-site data with identical variables and modeling methods. Four CI-D models were assessed for predicting care locations >6–12 hours, >12–18 hours, >18–24 hours, and >24–30 hours in the future. Design: Prognostic study comparing multi-institutional CI-D models’ performance in a single-site electronic health record dataset to an institution-specific CI-D model developed using identical variables and modelling methods. The institution did not participate in the multi-institutional dataset. Participants: All pediatric inpatients admitted from January 1st 2018 –February 29th 2020 through the emergency department. Main outcome(s) and measure(s): The main outcome was inpatient care in routine or ICU care locations. Results: A total of 29,037 pediatric hospital admissions were included, with 5,563 (19.2%) admitted directly to the ICU, 869 (3.0%) transferred from routine to ICU care, and 5,023 (17.3%) transferred from ICU to routine care. Patients had a median [IQR] age 68 months (15–157), 47.5% were female and 43.4% were black. The area under the receiver operating characteristic curve (AUROC) for the multi-institutional CI-D models applied to a single-site test dataset was 0.493–0.545 and area under the precision-recall curve (AUPRC) was 0.262–0.299. The single-site CI-D models applied to an independent single-site test dataset had an AUROC 0.906–0.944 and AUPRC range from 0.754–0.824. Accuracy at 0.95 sensitivity for those transferred from routine to ICU care was 72.6%-81.0%. Accuracy at 0.95 specificity was 58.2%-76.4% for patients who transferred from ICU to routine care. Conclusion and relevance: Models developed from multi-institutional datasets and intended for application to individual institutions should be assessed locally and may benefit from re-development with site-specific data prior to deployment.

Suggested Citation

  • Anita K Patel & Eduardo Trujillo-Rivera & James M Chamberlain & Hiroki Morizono & Murray M Pollack, 2024. "External evaluation of the Dynamic Criticality Index: A machine learning model to predict future need for ICU care in hospitalized pediatric patients," PLOS ONE, Public Library of Science, vol. 19(1), pages 1-17, January.
    • Handle: RePEc:plo:pone00:0288233
    DOI: 10.1371/journal.pone.0288233
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0288233
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0288233&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0288233?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Takaya Saito & Marc Rehmsmeier, 2015. "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets," PLOS ONE, Public Library of Science, vol. 10(3), pages 1-21, March.
    2. Devan V. Mehrotra & Ivan S. F. Chan & Roger L. Berger, 2003. "A Cautionary Note on Exact Unconditional Inference for a Difference between Two Independent Binomial Proportions," Biometrics, The International Biometric Society, vol. 59(2), pages 441-450, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Christopher J Greenwood & George J Youssef & Primrose Letcher & Jacqui A Macdonald & Lauryn J Hagg & Ann Sanson & Jenn Mcintosh & Delyse M Hutchinson & John W Toumbourou & Matthew Fuller-Tyszkiewicz &, 2020. "A comparison of penalised regression methods for informing the selection of predictive markers," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-14, November.
    2. Le, Hong Hanh & Viviani, Jean-Laurent, 2018. "Predicting bank failure: An improvement by implementing a machine-learning approach to classical financial ratios," Research in International Business and Finance, Elsevier, vol. 44(C), pages 16-25.
    3. João Chang Junior & Fábio Binuesa & Luiz Fernando Caneo & Aida Luiza Ribeiro Turquetto & Elisandra Cristina Trevisan Calvo Arita & Aline Cristina Barbosa & Alfredo Manoel da Silva Fernandes & Evelinda, 2020. "Improving preoperative risk-of-death prediction in surgery congenital heart defects using artificial intelligence model: A pilot study," PLOS ONE, Public Library of Science, vol. 15(9), pages 1-21, September.
    4. Arthur De Sá Ferreira & Ney Meziat-Filho & Ana Paula Antunes Ferreira, 2021. "Double threshold receiver operating characteristic plot for three-modal continuous predictors," Computational Statistics, Springer, vol. 36(3), pages 2231-2245, September.
    5. Chris J. Lloyd, 2010. "Bootstrap and Second-Order Tests of Risk Difference," Biometrics, The International Biometric Society, vol. 66(3), pages 975-982, September.
    6. Masabho P Milali & Samson S Kiware & Nicodem J Govella & Fredros Okumu & Naveen Bansal & Serdar Bozdag & Jacques D Charlwood & Marta F Maia & Sheila B Ogoma & Floyd E Dowell & George F Corliss & Maggy, 2020. "An autoencoder and artificial neural network-based method to estimate parity status of wild mosquitoes from near-infrared spectra," PLOS ONE, Public Library of Science, vol. 15(6), pages 1-16, June.
    7. Kajal Lahiri & Cheng Yang, 2023. "ROC and PRC Approaches to Evaluate Recession Forecasts," Journal of Business Cycle Research, Springer;Centre for International Research on Economic Tendency Surveys (CIRET), vol. 19(2), pages 119-148, September.
    8. Tzu-Hsuan Lin & Jehn-Ruey Jiang, 2021. "Credit Card Fraud Detection with Autoencoder and Probabilistic Random Forest," Mathematics, MDPI, vol. 9(21), pages 1-16, October.
    9. Felix Holzmeister & Magnus Johannesson & Colin F. Camerer & Yiling Chen & Teck-Hua Ho & Suzanne Hoogeveen & Juergen Huber & Noriko Imai & Taisuke Imai & Lawrence Jin & Michael Kirchler & Alexander Ly , 2025. "Examining the replicability of online experiments selected by a decision market," Nature Human Behaviour, Nature, vol. 9(2), pages 316-330, February.
    10. Apostolos Giannoulidis & Anastasios Gounaris & Athanasios Naskos & Nikodimos Nikolaidis & Daniel Caljouw, 2025. "Engineering and evaluating an unsupervised predictive maintenance solution: a cold-forming press case-study," Journal of Intelligent Manufacturing, Springer, vol. 36(3), pages 2121-2139, March.
    11. Dorian Knoblauch & Jürgen Großmann, 2023. "Towards a Risk-Based Continuous Auditing-Based Certification for Machine Learning," The Review of Socionetwork Strategies, Springer, vol. 17(2), pages 255-273, October.
    12. Alfred Krzywicki & David Muchlinski & Benjamin E. Goldsmith & Arcot Sowmya, 2022. "From academia to policy makers: a methodology for real-time forecasting of infrequent events," Journal of Computational Social Science, Springer, vol. 5(2), pages 1489-1510, November.
    13. Dueñas, Marco & Ortiz, Víctor & Riccaboni, Massimo & Serti, Francesco, 2021. "Assessing the Impact of COVID-19 on Trade: a Machine Learning Counterfactual Analysis," Working papers 79, Red Investigadores de Economía.
    14. Beibei Xu & Claira R. Seely & Tapomayukh Bhattacharjee & Taika von Konigslow, 2025. "Detection of Neonatal Calf Diarrhea Using Suckle Pressure and Machine Learning," Agriculture, MDPI, vol. 15(17), pages 1-18, August.
    15. Wei-Hsuan Lo-Ciganic & Julie M Donohue & Eric G Hulsey & Susan Barnes & Yuan Li & Courtney C Kuza & Qingnan Yang & Jeanine Buchanich & James L Huang & Christina Mair & Debbie L Wilson & Walid F Gellad, 2021. "Integrating human services and criminal justice data with claims data to predict risk of opioid overdose among Medicaid beneficiaries: A machine-learning approach," PLOS ONE, Public Library of Science, vol. 16(3), pages 1-18, March.
    16. Nica-Avram, Georgiana & Harvey, John & Smith, Gavin & Smith, Andrew & Goulding, James, 2021. "Identifying food insecurity in food sharing networks via machine learning," Journal of Business Research, Elsevier, vol. 131(C), pages 469-484.
    17. Ali J. Ghandour & Huda Hammoud & Samar Al-Hajj, 2020. "Analyzing Factors Associated with Fatal Road Crashes: A Machine Learning Approach," IJERPH, MDPI, vol. 17(11), pages 1-13, June.
    18. Song, Kwonsik & Anderson, Kyle & Lee, SangHyun, 2020. "An energy-cyber-physical system for personalized normative messaging interventions: Identification and classification of behavioral reference groups," Applied Energy, Elsevier, vol. 260(C).
    19. Soumadeep Saha & Utpal Garain & Arijit Ukil & Arpan Pal & Sundeep Khandelwal, 2023. "MedTric : A clinically applicable metric for evaluation of multi-label computational diagnostic systems," PLOS ONE, Public Library of Science, vol. 18(8), pages 1-19, August.
    20. Fisnik Doko & Slobodan Kalajdziski & Igor Mishkovski, 2021. "Credit Risk Model Based on Central Bank Credit Registry Data," JRFM, MDPI, vol. 14(3), pages 1-17, March.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0288233. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.