IDEAS home Printed from https://ideas.repec.org/a/plo/pcbi00/1010968.html
   My bibliography  Save this article

Validation framework for epidemiological models with application to COVID-19 models

Author

Listed:
  • Kimberly A Dautel
  • Ephraim Agyingi
  • Pras Pathmanathan

Abstract

Mathematical models have been an important tool during the COVID-19 pandemic, for example to predict demand of critical resources such as medical devices, personal protective equipment and diagnostic tests. Many COVID-19 models have been developed. However, there is relatively little information available regarding reliability of model predictions. Here we present a general model validation framework for epidemiological models focused around predictive capability for questions relevant to decision-making end-users. COVID-19 models are typically comprised of multiple releases, and provide predictions for multiple localities, and these characteristics are systematically accounted for in the framework, which is based around a set of validation scores or metrics that quantify model accuracy of specific quantities of interest including: date of peak, magnitude of peak, rate of recovery, and monthly cumulative counts. We applied the framework to retrospectively assess accuracy of death predictions for four COVID-19 models, and accuracy of hospitalization predictions for one COVID-19 model (models for which sufficient data was publicly available). When predicting date of peak deaths, the most accurate model had errors of approximately 15 days or less, for releases 3-6 weeks in advance of the peak. Death peak magnitude relative errors were generally in the 50% range 3-6 weeks before peak. Hospitalization predictions were less accurate than death predictions. All models were highly variable in predictive accuracy across regions. Overall, our framework provides a wealth of information on the predictive accuracy of epidemiological models and could be used in future epidemics to evaluate new models or support existing modeling methodologies, and thereby aid in informed model-based public health decision making. The code for the validation framework is available at https://doi.org/10.5281/zenodo.7102854.Author summary: During the COVID-19 pandemic many mathematical models have been developed. These mathematical models provide forecasts of key quantities such as number of infectious cases, number of hospitalizations, and number of deaths, in the upcoming weeks in the locality of interest. However, the reliability of model predictions is unclear. Currently, there have been few techniques employed to validate the performance of COVID-19 models that have focused on quantities that are especially of interest to end-users, such as when deaths will peak or the magnitude of the peak. Here, we provide an epidemiological model validation framework focused on questions relevant to decision-makers that utilize COVID-19 model predictions. We analyze four COVID-19 models with our framework, examining the accuracy of each model in predicting the date and magnitude of a peak, recovery rate, and monthly cumulative counts, for predictions of deaths and of hospitalizations. Our results show that the mathematical models produce highly variable predictions across regions. Our framework demonstrates the need for predictive reliability of epidemiological models.

Suggested Citation

  • Kimberly A Dautel & Ephraim Agyingi & Pras Pathmanathan, 2023. "Validation framework for epidemiological models with application to COVID-19 models," PLOS Computational Biology, Public Library of Science, vol. 19(3), pages 1-24, March.
  • Handle: RePEc:plo:pcbi00:1010968
    DOI: 10.1371/journal.pcbi.1010968
    as

    Download full text from publisher

    File URL: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1010968
    Download Restriction: no

    File URL: https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1010968&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pcbi.1010968?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Ndaïrou, Faïçal & Area, Iván & Nieto, Juan J. & Torres, Delfim F.M., 2020. "Mathematical modeling of COVID-19 transmission dynamics with a case study of Wuhan," Chaos, Solitons & Fractals, Elsevier, vol. 135(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Masum, Mohammad & Masud, M.A. & Adnan, Muhaiminul Islam & Shahriar, Hossain & Kim, Sangil, 2022. "Comparative study of a mathematical epidemic model, statistical modeling, and deep learning for COVID-19 forecasting and management," Socio-Economic Planning Sciences, Elsevier, vol. 80(C).
    2. Basnarkov, Lasko, 2021. "SEAIR Epidemic spreading model of COVID-19," Chaos, Solitons & Fractals, Elsevier, vol. 142(C).
    3. Cooper, Ian & Mondal, Argha & Antonopoulos, Chris G., 2020. "Dynamic tracking with model-based forecasting for the spread of the COVID-19 pandemic," Chaos, Solitons & Fractals, Elsevier, vol. 139(C).
    4. Cooper, Ian & Mondal, Argha & Antonopoulos, Chris G., 2020. "A SIR model assumption for the spread of COVID-19 in different communities," Chaos, Solitons & Fractals, Elsevier, vol. 139(C).
    5. Mishra, A.M. & Purohit, S.D. & Owolabi, K.M. & Sharma, Y.D., 2020. "A nonlinear epidemiological model considering asymptotic and quarantine classes for SARS CoV-2 virus," Chaos, Solitons & Fractals, Elsevier, vol. 138(C).
    6. Memon, Zaibunnisa & Qureshi, Sania & Memon, Bisharat Rasool, 2021. "Assessing the role of quarantine and isolation as control strategies for COVID-19 outbreak: A case study," Chaos, Solitons & Fractals, Elsevier, vol. 144(C).
    7. Amaral, Marco A. & Oliveira, Marcelo M. de & Javarone, Marco A., 2021. "An epidemiological model with voluntary quarantine strategies governed by evolutionary game dynamics," Chaos, Solitons & Fractals, Elsevier, vol. 143(C).
    8. Ullah, Mohammad Sharif & Higazy, M. & Kabir, K.M. Ariful, 2022. "Dynamic analysis of mean-field and fractional-order epidemic vaccination strategies by evolutionary game approach," Chaos, Solitons & Fractals, Elsevier, vol. 162(C).
    9. Rafiq, Danish & Suhail, Suhail Ahmad & Bazaz, Mohammad Abid, 2020. "Evaluation and prediction of COVID-19 in India: A case study of worst hit states," Chaos, Solitons & Fractals, Elsevier, vol. 139(C).
    10. Faïçal Ndaïrou & Iván Area & Delfim F. M. Torres, 2020. "Mathematical Modeling of Japanese Encephalitis under Aquatic Environmental Effects," Mathematics, MDPI, vol. 8(11), pages 1-14, October.
    11. Swapnarekha, H. & Behera, Himansu Sekhar & Nayak, Janmenjoy & Naik, Bighnaraj, 2020. "Role of intelligent computing in COVID-19 prognosis: A state-of-the-art review," Chaos, Solitons & Fractals, Elsevier, vol. 138(C).
    12. Li, Tingting & Guo, Youming, 2022. "Optimal control and cost-effectiveness analysis of a new COVID-19 model for Omicron strain," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 606(C).
    13. Florian Dorn & Sahamoddin Khailaie & Marc Stoeckli & Sebastian C. Binder & Tanmay Mitra & Berit Lange & Stefan Lautenbacher & Andreas Peichl & Patrizio Vanella & Timo Wollmershäuser & Clemens Fuest & , 2023. "The common interests of health protection and the economy: evidence from scenario calculations of COVID-19 containment policies," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 24(1), pages 67-74, February.
    14. Aldila, Dipo & Khoshnaw, Sarbaz H.A. & Safitri, Egi & Anwar, Yusril Rais & Bakry, Aanisah R.Q. & Samiadji, Brenda M. & Anugerah, Demas A. & GH, M. Farhan Alfarizi & Ayulani, Indri D. & Salim, Sheryl N, 2020. "A mathematical study on the spread of COVID-19 considering social distancing and rapid assessment: The case of Jakarta, Indonesia," Chaos, Solitons & Fractals, Elsevier, vol. 139(C).
    15. Rabih Ghostine & Mohamad Gharamti & Sally Hassrouny & Ibrahim Hoteit, 2021. "Mathematical Modeling of Immune Responses against SARS-CoV-2 Using an Ensemble Kalman Filter," Mathematics, MDPI, vol. 9(19), pages 1-13, September.
    16. Koutsellis, Themistoklis & Nikas, Alexandros, 2020. "A predictive model and country risk assessment for COVID-19: An application of the Limited Failure Population concept," Chaos, Solitons & Fractals, Elsevier, vol. 140(C).
    17. Ahumada, M. & Ledesma-Araujo, A. & Gordillo, L. & Marín, J.F., 2023. "Mutation and SARS-CoV-2 strain competition under vaccination in a modified SIR model," Chaos, Solitons & Fractals, Elsevier, vol. 166(C).
    18. Usama H. Issa & Ashraf Balabel & Mohammed Abdelhakeem & Medhat M. A. Osman, 2021. "Developing a Risk Model for Assessment and Control of the Spread of COVID-19," Risks, MDPI, vol. 9(2), pages 1-15, February.
    19. Iván Area & Henrique Lorenzo & Pedro J. Marcos & Juan J. Nieto, 2021. "One Year of the COVID-19 Pandemic in Galicia: A Global View of Age-Group Statistics during Three Waves," IJERPH, MDPI, vol. 18(10), pages 1-14, May.
    20. Caraballo, T. & Settati, A. & Lahrouz, A. & Boutouil, S. & Harchaoui, B., 2024. "On the stochastic threshold of the COVID-19 epidemic model incorporating jump perturbations," Chaos, Solitons & Fractals, Elsevier, vol. 180(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pcbi00:1010968. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ploscompbiol (email available below). General contact details of provider: https://journals.plos.org/ploscompbiol/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.