IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v39y2023i3p1366-1383.html
   My bibliography  Save this article

Comparing trained and untrained probabilistic ensemble forecasts of COVID-19 cases and deaths in the United States

Author

Listed:
  • Ray, Evan L.
  • Brooks, Logan C.
  • Bien, Jacob
  • Biggerstaff, Matthew
  • Bosse, Nikos I.
  • Bracher, Johannes
  • Cramer, Estee Y.
  • Funk, Sebastian
  • Gerding, Aaron
  • Johansson, Michael A.
  • Rumack, Aaron
  • Wang, Yijin
  • Zorn, Martha
  • Tibshirani, Ryan J.
  • Reich, Nicholas G.

Abstract

The U.S. COVID-19 Forecast Hub aggregates forecasts of the short-term burden of COVID-19 in the United States from many contributing teams. We study methods for building an ensemble that combines forecasts from these teams. These experiments have informed the ensemble methods used by the Hub. To be most useful to policymakers, ensemble forecasts must have stable performance in the presence of two key characteristics of the component forecasts: (1) occasional misalignment with the reported data, and (2) instability in the relative performance of component forecasters over time. Our results indicate that in the presence of these challenges, an untrained and robust approach to ensembling using an equally weighted median of all component forecasts is a good choice to support public health decision-makers. In settings where some contributing forecasters have a stable record of good performance, trained ensembles that give those forecasters higher weight can also be helpful.

Suggested Citation

  • Ray, Evan L. & Brooks, Logan C. & Bien, Jacob & Biggerstaff, Matthew & Bosse, Nikos I. & Bracher, Johannes & Cramer, Estee Y. & Funk, Sebastian & Gerding, Aaron & Johansson, Michael A. & Rumack, Aaron, 2023. "Comparing trained and untrained probabilistic ensemble forecasts of COVID-19 cases and deaths in the United States," International Journal of Forecasting, Elsevier, vol. 39(3), pages 1366-1383.
    • Handle: RePEc:eee:intfor:v:39:y:2023:i:3:p:1366-1383
    DOI: 10.1016/j.ijforecast.2022.06.005
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0169207022000966
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ijforecast.2022.06.005?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Evan L Ray & Nicholas G Reich, 2018. "Prediction of infectious disease epidemics via weighted density ensembles," PLOS Computational Biology, Public Library of Science, vol. 14(2), pages 1-23, February.
    2. Nicholas G Reich & Stephen A Lauer & Krzysztof Sakrejda & Sopon Iamsirithaworn & Soawapak Hinjoy & Paphanij Suangtho & Suthanun Suthachana & Hannah E Clapham & Henrik Salje & Derek A T Cummings & Just, 2016. "Challenges in Real-Time Prediction of Infectious Disease: A Case Study of Dengue in Thailand," PLOS Neglected Tropical Diseases, Public Library of Science, vol. 10(6), pages 1-17, June.
    3. Stephen C. Hora & Benjamin R. Fransen & Natasha Hawkins & Irving Susel, 2013. "Median Aggregation of Distribution Functions," Decision Analysis, INFORMS, vol. 10(4), pages 279-291, December.
    4. Nicholas G Reich & Craig J McGowan & Teresa K Yamana & Abhinav Tushar & Evan L Ray & Dave Osthus & Sasikiran Kandula & Logan C Brooks & Willow Crawford-Crudell & Graham Casey Gibson & Evan Moore & Reb, 2019. "Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S," PLOS Computational Biology, Public Library of Science, vol. 15(11), pages 1-19, November.
    5. Felipe J Colón-González & Leonardo Soares Bastos & Barbara Hofmann & Alison Hopkin & Quillon Harpham & Tom Crocker & Rosanna Amato & Iacopo Ferrario & Francesca Moschini & Samuel James & Sajni Malde &, 2021. "Probabilistic seasonal dengue forecasting in Vietnam: A modelling study using superensembles," PLOS Medicine, Public Library of Science, vol. 18(3), pages 1-30, March.
    6. Fabio Busetti, 2017. "Quantile Aggregation of Density Forecasts," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 79(4), pages 495-512, August.
    7. Claeskens, Gerda & Magnus, Jan R. & Vasnev, Andrey L. & Wang, Wendun, 2016. "The forecast combination puzzle: A simple theoretical explanation," International Journal of Forecasting, Elsevier, vol. 32(3), pages 754-762.
    8. Johannes Bracher & Evan L Ray & Tilmann Gneiting & Nicholas G Reich, 2021. "Evaluating epidemic forecasts in an interval format," PLOS Computational Biology, Public Library of Science, vol. 17(2), pages 1-15, February.
    9. Roopesh Ranjan & Tilmann Gneiting, 2010. "Combining probability forecasts," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 72(1), pages 71-91, January.
    10. Simon Pollett & Michael A Johansson & Nicholas G Reich & David Brett-Major & Sara Y Del Valle & Srinivasan Venkatramanan & Rachel Lowe & Travis Porco & Irina Maljkovic Berry & Alina Deshpande & Moritz, 2021. "Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines," PLOS Medicine, Public Library of Science, vol. 18(10), pages 1-12, October.
    11. Logan C Brooks & David C Farrow & Sangwon Hyun & Ryan J Tibshirani & Roni Rosenfeld, 2018. "Nonmechanistic forecasts of seasonal influenza with iterative one-week-ahead distributions," PLOS Computational Biology, Public Library of Science, vol. 14(6), pages 1-29, June.
    12. Anil Gaba & Ilia Tsetlin & Robert L. Winkler, 2017. "Combining Interval Forecasts," Decision Analysis, INFORMS, vol. 14(1), pages 1-20, March.
    13. James Turtle & Pete Riley & Michal Ben-Nun & Steven Riley, 2021. "Accurate influenza forecasts using type-specific incidence data for small geographic units," PLOS Computational Biology, Public Library of Science, vol. 17(7), pages 1-20, July.
    14. A. Agosto & Alexandra Campmas & P. Giudici & A. Renda, 2021. "Monitoring COVID-19 contagion growth," Post-Print hal-03407115, HAL.
    15. Tilmann Gneiting & Fadoua Balabdaoui & Adrian E. Raftery, 2007. "Probabilistic forecasts, calibration and sharpness," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 69(2), pages 243-268, April.
    16. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    17. Dave Osthus & Kelly R. Moran, 2021. "Multiscale influenza forecasting," Nature Communications, Nature, vol. 12(1), pages 1-11, December.
    18. Kenneth C. Lichtendahl & Yael Grushka-Cockayne & Robert L. Winkler, 2013. "Is It Better to Average Probabilities or Quantiles?," Management Science, INFORMS, vol. 59(7), pages 1594-1611, July.
    19. J. Bracher & D. Wolffram & J. Deuschel & K. Görgen & J. L. Ketterer & A. Ullrich & S. Abbott & M. V. Barbarossa & D. Bertsimas & S. Bhatia & M. Bodych & N. I. Bosse & J. P. Burgard & L. Castro & G. Fa, 2021. "A pre-registered short-term forecasting study of COVID-19 in Germany and Poland during the second wave," Nature Communications, Nature, vol. 12(1), pages 1-16, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    2. Taylor, James W. & Taylor, Kathryn S., 2023. "Combining probabilistic forecasts of COVID-19 mortality in the United States," European Journal of Operational Research, Elsevier, vol. 304(1), pages 25-41.
    3. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    4. Kathryn S Taylor & James W Taylor, 2022. "Interval forecasts of weekly incident and cumulative COVID-19 mortality in the United States: A comparison of combining methods," PLOS ONE, Public Library of Science, vol. 17(3), pages 1-25, March.
    5. Malte Knüppel & Fabian Krüger, 2022. "Forecast uncertainty, disagreement, and the linear pool," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(1), pages 23-41, January.
    6. Junyi Lu & Sebastian Meyer, 2020. "Forecasting Flu Activity in the United States: Benchmarking an Endemic-Epidemic Beta Model," IJERPH, MDPI, vol. 17(4), pages 1-13, February.
    7. Martin, Gael M. & Loaiza-Maya, Rubén & Maneesoonthorn, Worapree & Frazier, David T. & Ramírez-Hassan, Andrés, 2022. "Optimal probabilistic forecasts: When do they work?," International Journal of Forecasting, Elsevier, vol. 38(1), pages 384-406.
    8. Stephen Hora & Erim Kardeş, 2015. "Calibration, sharpness and the weighting of experts in a linear opinion pool," Annals of Operations Research, Springer, vol. 229(1), pages 429-450, June.
    9. Robert L. Winkler & Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose, 2019. "Probability Forecasts and Their Combination: A Research Perspective," Decision Analysis, INFORMS, vol. 16(4), pages 239-260, December.
    10. Yang, Dazhi & van der Meer, Dennis, 2021. "Post-processing in solar forecasting: Ten overarching thinking tools," Renewable and Sustainable Energy Reviews, Elsevier, vol. 140(C).
    11. Makariou, Despoina & Barrieu, Pauline & Tzougas, George, 2021. "A finite mixture modelling perspective for combining experts’ opinions with an application to quantile-based risk measures," LSE Research Online Documents on Economics 110763, London School of Economics and Political Science, LSE Library.
    12. Despoina Makariou & Pauline Barrieu & George Tzougas, 2021. "A Finite Mixture Modelling Perspective for Combining Experts’ Opinions with an Application to Quantile-Based Risk Measures," Risks, MDPI, vol. 9(6), pages 1-25, June.
    13. Knut Are Aastveit & James Mitchell & Francesco Ravazzolo & Herman van Dijk, 2018. "The Evolution of Forecast Density Combinations in Economics," Tinbergen Institute Discussion Papers 18-069/III, Tinbergen Institute.
    14. Lahiri, Kajal & Yang, Liu, 2013. "Forecasting Binary Outcomes," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1025-1106, Elsevier.
    15. Satopää, Ville A. & Baron, Jonathan & Foster, Dean P. & Mellers, Barbara A. & Tetlock, Philip E. & Ungar, Lyle H., 2014. "Combining multiple probability predictions using a simple logit model," International Journal of Forecasting, Elsevier, vol. 30(2), pages 344-356.
    16. Ruben Loaiza‐Maya & Gael M. Martin & David T. Frazier, 2021. "Focused Bayesian prediction," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 36(5), pages 517-543, August.
    17. Fabian Kruger & Hendrik Plett, 2022. "Prediction intervals for economic fixed-event forecasts," Papers 2210.13562, arXiv.org, revised Mar 2024.
    18. Tommaso Proietti & Martyna Marczak & Gianluigi Mazzi, 2017. "Euromind‐ D : A Density Estimate of Monthly Gross Domestic Product for the Euro Area," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(3), pages 683-703, April.
    19. Coroneo, Laura & Iacone, Fabrizio & Paccagnini, Alessia & Santos Monteiro, Paulo, 2023. "Testing the predictive accuracy of COVID-19 forecasts," International Journal of Forecasting, Elsevier, vol. 39(2), pages 606-622.
    20. Anil Gaba & Ilia Tsetlin & Robert L. Winkler, 2017. "Combining Interval Forecasts," Decision Analysis, INFORMS, vol. 14(1), pages 1-20, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:39:y:2023:i:3:p:1366-1383. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.