IDEAS home Printed from https://ideas.repec.org/a/wly/japmet/v38y2023i4p596-622.html
   My bibliography  Save this article

Robust forecast superiority testing with an application to assessing pools of expert forecasters

Author

Listed:
  • Valentina Corradi
  • Sainan Jin
  • Norman R. Swanson

Abstract

We develop forecast superiority tests that are robust to the choice of loss function by following Jin, Corradi and Swanson (JCS: 2017), and relying on a mapping between generic loss forecast evaluation and stochastic dominance principles. However, unlike JCS tests, which are not uniformly valid and are correctly sized only under the least favorable case, our tests are uniformly asymptotically valid and non‐conservative. To show this, we establish uniform convergence of HAC variance estimators. Monte Carlo experiments indicate good finite sample performance of our tests, and an empirical illustration suggests that prior forecast accuracy matters in the Survey of Professional Forecasters.

Suggested Citation

  • Valentina Corradi & Sainan Jin & Norman R. Swanson, 2023. "Robust forecast superiority testing with an application to assessing pools of expert forecasters," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(4), pages 596-622, June.
  • Handle: RePEc:wly:japmet:v:38:y:2023:i:4:p:596-622
    DOI: 10.1002/jae.2962
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/jae.2962
    Download Restriction: no

    File URL: https://libkey.io/10.1002/jae.2962?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Linton, Oliver & Song, Kyungchul & Whang, Yoon-Jae, 2010. "An improved bootstrap test of stochastic dominance," Journal of Econometrics, Elsevier, vol. 154(2), pages 186-202, February.
    2. Victor Zarnowitz & Phillip Braun, 1993. "Twenty-two Years of the NBER-ASA Quarterly Economic Outlook Surveys: Aspects and Comparisons of Forecasting Performance," NBER Chapters, in: Business Cycles, Indicators, and Forecasting, pages 11-94, National Bureau of Economic Research, Inc.
    3. Andrews, Donald W.K. & Shi, Xiaoxia, 2017. "Inference based on many conditional moment inequalities," Journal of Econometrics, Elsevier, vol. 196(2), pages 275-287.
    4. Graham Elliott & Allan Timmermann & Ivana Komunjer, 2005. "Estimation and Testing of Forecast Rationality under Flexible Loss," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 72(4), pages 1107-1125.
    5. Timmermann, Allan, 2006. "Forecast Combinations," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 4, pages 135-196, Elsevier.
    6. Francis X. Diebold & Minchul Shin, 2017. "Assessing point forecast accuracy by stochastic error distance," Econometric Reviews, Taylor & Francis Journals, vol. 36(6-9), pages 588-598, October.
    7. Andrew J. Patton, 2020. "Comparing Possibly Misspecified Forecasts," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 38(4), pages 796-809, October.
    8. Donald W. K. Andrews & Xiaoxia Shi, 2013. "Inference Based on Conditional Moment Inequalities," Econometrica, Econometric Society, vol. 81(2), pages 609-666, March.
    9. Capistrán, Carlos & Timmermann, Allan, 2009. "Forecast Combination With Entry and Exit of Experts," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(4), pages 428-440.
    10. Diebold, Francis X. & Shin, Minchul, 2015. "Assessing point forecast accuracy by stochastic loss distance," Economics Letters, Elsevier, vol. 130(C), pages 37-38.
    11. Genre, Véronique & Kenny, Geoff & Meyler, Aidan & Timmermann, Allan, 2013. "Combining expert forecasts: Can anything beat the simple average?," International Journal of Forecasting, Elsevier, vol. 29(1), pages 108-121.
    12. Laura Coroneo & Valentina Corradi & Paulo Santos Monteiro, 2018. "Testing for optimal monetary policy via moment inequalities," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 33(6), pages 780-796, September.
    13. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    14. Sander Barendse & Andrew J. Patton, 2022. "Comparing Predictive Accuracy in the Presence of a Loss Function Shape Parameter," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 40(3), pages 1057-1069, June.
    15. Raffaella Giacomini & Barbara Rossi, 2009. "Detecting and Predicting Forecast Breakdowns," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 76(2), pages 669-705.
    16. repec:cwl:cwldpp:1840rr is not listed on IDEAS
    17. Graham Elliott & Ivana Komunjer & Allan Timmermann, 2008. "Biases in Macroeconomic Forecasts: Irrationality or Asymmetric Loss?," Journal of the European Economic Association, MIT Press, vol. 6(1), pages 122-157, March.
    18. Bierens, Herman J., 1982. "Consistent model specification tests," Journal of Econometrics, Elsevier, vol. 20(1), pages 105-134, October.
    19. Hansen, Bruce E., 2008. "Uniform Convergence Rates For Kernel Estimation With Dependent Data," Econometric Theory, Cambridge University Press, vol. 24(3), pages 726-748, June.
    20. Fair, Ray C & Shiller, Robert J, 1990. "Comparing Information in Forecasts from Econometric Models," American Economic Review, American Economic Association, vol. 80(3), pages 375-389, June.
    21. Arvanitis, Stelios & Post, Thierry & Potì, Valerio & Karabati, Selcuk, 2021. "Nonparametric tests for Optimal Predictive Ability," International Journal of Forecasting, Elsevier, vol. 37(2), pages 881-898.
    22. Newey, Whitney & West, Kenneth, 2014. "A simple, positive semi-definite, heteroscedasticity and autocorrelation consistent covariance matrix," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 33(1), pages 125-132.
    23. Andrews, Donald W.K. & Shi, Xiaoxia, 2014. "Nonparametric inference based on conditional moment inequalities," Journal of Econometrics, Elsevier, vol. 179(1), pages 31-45.
    24. Lahiri, Kajal & Peng, Huaming & Zhao, Yongchen, 2015. "Testing the value of probability forecasts for calibrated combining," International Journal of Forecasting, Elsevier, vol. 31(1), pages 113-129.
    25. Valentina Corradi & Norman Swanson, 2013. "A Survey of Recent Advances in Forecast Accuracy Comparison Testing, with an Extension to Stochastic Dominance," Departmental Working Papers 201309, Rutgers University, Department of Economics.
    26. Li, Jia & Liao, Zhipeng, 2020. "Uniform nonparametric inference for time series," Journal of Econometrics, Elsevier, vol. 219(1), pages 38-51.
    27. Valentina Corradi & Norman R. Swanson, 2007. "Nonparametric Bootstrap Procedures For Predictive Inference Based On Recursive Estimation Schemes," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 48(1), pages 67-109, February.
    28. Gneiting, Tilmann, 2011. "Making and Evaluating Point Forecasts," Journal of the American Statistical Association, American Statistical Association, vol. 106(494), pages 746-762.
    29. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    30. Kajal Lahiri & Huaming Peng & Yongchen Zhao, 2017. "Online learning and forecast combination in unbalanced panels," Econometric Reviews, Taylor & Francis Journals, vol. 36(1-3), pages 257-288, March.
    31. Swanson, Norman R. & White, Halbert, 1997. "Forecasting economic time series using flexible versus fixed specification and linear versus nonlinear econometric models," International Journal of Forecasting, Elsevier, vol. 13(4), pages 439-461, December.
    32. Patton, Andrew J. & Timmermann, Allan, 2007. "Testing Forecast Optimality Under Unknown Loss," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 1172-1184, December.
    33. Xiaohong Chen & Norman R. Swanson (ed.), 2013. "Recent Advances and Future Directions in Causality, Prediction, and Specification Analysis," Springer Books, Springer, edition 127, number 978-1-4614-1653-1, November.
    34. Donald W. K. Andrews & Gustavo Soares, 2010. "Inference for Parameters Defined by Moment Inequalities Using Generalized Moment Selection," Econometrica, Econometric Society, vol. 78(1), pages 119-157, January.
    35. Jia Li & Zhipeng Liao & Mengsi Gao, 2020. "Uniform nonparametric inference for time series using Stata," Stata Journal, StataCorp LP, vol. 20(3), pages 706-720, September.
    36. Jin, Sainan & Corradi, Valentina & Swanson, Norman R., 2017. "Robust Forecast Comparison," Econometric Theory, Cambridge University Press, vol. 33(6), pages 1306-1351, December.
    37. Pablo Pincheira & Nicolás Hardy & Felipe Muñoz, 2021. "“Go Wild for a While!”: A New Test for Forecast Evaluation in Nested Models," Mathematics, MDPI, vol. 9(18), pages 1-28, September.
    38. Pincheira, Pablo & Hardy, Nicolás & Muñoz, Felipe, 2021. ""Go wild for a while!": A new asymptotically Normal test for forecast evaluation in nested models," MPRA Paper 105368, University Library of Munich, Germany.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jin, Sainan & Corradi, Valentina & Swanson, Norman R., 2017. "Robust Forecast Comparison," Econometric Theory, Cambridge University Press, vol. 33(6), pages 1306-1351, December.
    2. Norman R. Swanson & Weiqi Xiong, 2018. "Big data analytics in economics: What have we learned so far, and where should we go from here?," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 51(3), pages 695-746, August.
    3. Graham Elliott & Allan Timmermann, 2016. "Forecasting in Economics and Finance," Annual Review of Economics, Annual Reviews, vol. 8(1), pages 81-110, October.
    4. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    5. Dimitriadis, Timo & Schnaitmann, Julie, 2021. "Forecast encompassing tests for the expected shortfall," International Journal of Forecasting, Elsevier, vol. 37(2), pages 604-621.
    6. Hsu, Yu-Chin & Shen, Shu, 2019. "Testing treatment effect heterogeneity in regression discontinuity designs," Journal of Econometrics, Elsevier, vol. 208(2), pages 468-486.
    7. Matteo Iacopini & Francesco Ravazzolo & Luca Rossini, 2020. "Proper scoring rules for evaluating asymmetry in density forecasting," Papers 2006.11265, arXiv.org, revised Sep 2020.
    8. Patrick Schmidt & Matthias Katzfuss & Tilmann Gneiting, 2021. "Interpretation of point forecasts with unknown directive," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 36(6), pages 728-743, September.
    9. Raffaella Giacomini & Barbara Rossi, 2013. "Forecasting in macroeconomics," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 17, pages 381-408, Edward Elgar Publishing.
    10. Yen, Yu-Min & Yen, Tso-Jung, 2021. "Testing forecast accuracy of expectiles and quantiles with the extremal consistent loss functions," International Journal of Forecasting, Elsevier, vol. 37(2), pages 733-758.
    11. Lahiri, Kajal & Yang, Liu, 2013. "Forecasting Binary Outcomes," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1025-1106, Elsevier.
    12. Szafranek, Karol, 2019. "Bagged neural networks for forecasting Polish (low) inflation," International Journal of Forecasting, Elsevier, vol. 35(3), pages 1042-1059.
    13. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    14. Francis X. Diebold & Minchul Shin, 2017. "Assessing point forecast accuracy by stochastic error distance," Econometric Reviews, Taylor & Francis Journals, vol. 36(6-9), pages 588-598, October.
    15. Xue Gong & Weiguo Zhang & Yuan Zhao & Xin Ye, 2023. "Forecasting stock volatility with a large set of predictors: A new forecast combination method," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(7), pages 1622-1647, November.
    16. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    17. Dichtl, Hubert & Drobetz, Wolfgang & Neuhierl, Andreas & Wendt, Viktoria-Sophie, 2021. "Data snooping in equity premium prediction," International Journal of Forecasting, Elsevier, vol. 37(1), pages 72-94.
    18. Jos Jansen & Jasper de Winter, 2016. "Improving model-based near-term GDP forecasts by subjective forecasts: A real-time exercise for the G7 countries," DNB Working Papers 507, Netherlands Central Bank, Research Department.
    19. Yoichi Arai & Yu‐Chin Hsu & Toru Kitagawa & Ismael Mourifié & Yuanyuan Wan, 2022. "Testing identifying assumptions in fuzzy regression discontinuity designs," Quantitative Economics, Econometric Society, vol. 13(1), pages 1-28, January.
    20. Daniel Borup & Jonas N. Eriksen & Mads M. Kjær & Martin Thyrsgaard, 2020. "Predicting bond return predictability," CREATES Research Papers 2020-09, Department of Economics and Business Economics, Aarhus University.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:japmet:v:38:y:2023:i:4:p:596-622. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.interscience.wiley.com/jpages/0883-7252/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.