IDEAS home Printed from https://ideas.repec.org/a/eee/econom/v237y2023i2s0304407621002256.html
   My bibliography  Save this article

Comparing forecasting performance in cross-sections

Author

Listed:
  • Qu, Ritong
  • Timmermann, Allan
  • Zhu, Yinchu

Abstract

This paper develops new methods for pairwise comparisons of predictive accuracy with cross-sectional data. Using a common factor setup, we establish conditions on cross-sectional dependencies in forecast errors which allow us to test the null of equal predictive accuracy on a single cross-section of forecasts. We consider both unconditional tests of equal predictive accuracy as well as tests that condition on the realization of common factors and show how to decompose forecast errors into exposures to common factors and idiosyncratic components. An empirical application compares the predictive accuracy of financial analysts’ short-term earnings forecasts across six brokerage firms.

Suggested Citation

  • Qu, Ritong & Timmermann, Allan & Zhu, Yinchu, 2023. "Comparing forecasting performance in cross-sections," Journal of Econometrics, Elsevier, vol. 237(2).
  • Handle: RePEc:eee:econom:v:237:y:2023:i:2:s0304407621002256
    DOI: 10.1016/j.jeconom.2021.02.011
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0304407621002256
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.jeconom.2021.02.011?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    2. Clark, Todd E. & McCracken, Michael W., 2001. "Tests of equal forecast accuracy and encompassing for nested models," Journal of Econometrics, Elsevier, vol. 105(1), pages 85-110, November.
    3. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    4. Bartosz Mackowiak & Mirko Wiederholt, 2009. "Optimal Sticky Prices under Rational Inattention," American Economic Review, American Economic Association, vol. 99(3), pages 769-803, June.
    5. Laura Liu & Hyungsik Roger Moon & Frank Schorfheide, 2020. "Forecasting With Dynamic Panel Data Models," Econometrica, Econometric Society, vol. 88(1), pages 171-201, January.
    6. Andrew Patton & Allan Timmermann, 2012. "Forecast Rationality Tests Based on Multi-Horizon Bounds," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(1), pages 1-17.
    7. Laura Liu & Hyungsik Roger Moon & Frank Schorfheide, 2023. "Forecasting with a panel Tobit model," Quantitative Economics, Econometric Society, vol. 14(1), pages 117-159, January.
    8. Peter Reinhard Hansen & Allan Timmermann, 2015. "Equivalence Between Out‐of‐Sample Forecast Comparisons and Wald Statistics," Econometrica, Econometric Society, vol. 83, pages 2485-2505, November.
    9. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    10. Graham Elliott & Allan Timmermann & Ivana Komunjer, 2005. "Estimation and Testing of Forecast Rationality under Flexible Loss," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 72(4), pages 1107-1125.
    11. Yock Y. Chong & David F. Hendry, 1986. "Econometric Evaluation of Linear Macro-Economic Models," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 53(4), pages 671-690.
    12. M. Hashem Pesaran, 2021. "General diagnostic tests for cross-sectional dependence in panels," Empirical Economics, Springer, vol. 60(1), pages 13-50, January.
    13. Seung C. Ahn & Alex R. Horenstein, 2013. "Eigenvalue Ratio Test for the Number of Factors," Econometrica, Econometric Society, vol. 81(3), pages 1203-1227, May.
    14. Alexei Onatski, 2009. "Testing Hypotheses About the Number of Factors in Large Factor Models," Econometrica, Econometric Society, vol. 77(5), pages 1447-1479, September.
    15. Davies, Anthony & Lahiri, Kajal, 1995. "A new framework for analyzing survey forecasts using three-dimensional panel data," Journal of Econometrics, Elsevier, vol. 68(1), pages 205-227, July.
    16. Keane, Michael P & Runkle, David E, 1990. "Testing the Rationality of Price Forecasts: New Evidence from Panel Data," American Economic Review, American Economic Association, vol. 80(4), pages 714-735, September.
    17. Timmermann, Allan & Qu, Ritong & Zhu, Yinchu, 2019. "Do Any Economists Have Superior Forecasting Skills?," CEPR Discussion Papers 14112, C.E.P.R. Discussion Papers.
    18. Xu Cheng & Zhipeng Liao & Frank Schorfheide, 2016. "Shrinkage Estimation of High-Dimensional Factor Models with Structural Instabilities," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 83(4), pages 1511-1543.
    19. Marcin Kacperczyk & Stijn Van Nieuwerburgh & Laura Veldkamp, 2014. "Time-Varying Fund Manager Skill," Journal of Finance, American Finance Association, vol. 69(4), pages 1455-1484, August.
    20. Jushan Bai, 2003. "Inferential Theory for Factor Models of Large Dimensions," Econometrica, Econometric Society, vol. 71(1), pages 135-171, January.
    21. M. Hashem Pesaran, 2006. "Estimation and Inference in Large Heterogeneous Panels with a Multifactor Error Structure," Econometrica, Econometric Society, vol. 74(4), pages 967-1012, July.
    22. David Blake & Alberto G. Rossi & Allan Timmermann & Ian Tonks & Russ Wermers, 2013. "Decentralized Investment Management: Evidence from the Pension Fund Industry," Journal of Finance, American Finance Association, vol. 68(3), pages 1133-1178, June.
    23. McCracken, Michael W., 2007. "Asymptotics for out of sample tests of Granger causality," Journal of Econometrics, Elsevier, vol. 140(2), pages 719-752, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Timmermann, Allan & Zhu, Yinchu, 2019. "Comparing Forecasting Performance with Panel Data," CEPR Discussion Papers 13746, C.E.P.R. Discussion Papers.
    2. Timmermann, Allan & Qu, Ritong & Zhu, Yinchu, 2019. "Do Any Economists Have Superior Forecasting Skills?," CEPR Discussion Papers 14112, C.E.P.R. Discussion Papers.
    3. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    4. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    5. Nathan Goldstein & Ben‐Zion Zilberfarb, 2023. "The closer we get, the better we are?," Economic Inquiry, Western Economic Association International, vol. 61(2), pages 364-376, April.
    6. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    7. Clark, Todd E. & West, Kenneth D., 2007. "Approximately normal tests for equal predictive accuracy in nested models," Journal of Econometrics, Elsevier, vol. 138(1), pages 291-311, May.
    8. Gonçalves, Sílvia & McCracken, Michael W. & Perron, Benoit, 2017. "Tests of equal accuracy for nested models with estimated factors," Journal of Econometrics, Elsevier, vol. 198(2), pages 231-252.
    9. Firmin Doko Tchatoka & Qazi Haque, 2023. "On bootstrapping tests of equal forecast accuracy for nested models," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(7), pages 1844-1864, November.
    10. Pincheira, Pablo & Hardy, Nicolas, 2022. "Correlation Based Tests of Predictability," MPRA Paper 112014, University Library of Munich, Germany.
    11. Raffaella Giacomini & Barbara Rossi, 2013. "Forecasting in macroeconomics," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 17, pages 381-408, Edward Elgar Publishing.
    12. Zhu, Yinchu & Timmermann, Allan, 2022. "Conditional rotation between forecasting models," Journal of Econometrics, Elsevier, vol. 231(2), pages 329-347.
    13. Busetti, Fabio & Marcucci, Juri, 2013. "Comparing forecast accuracy: A Monte Carlo investigation," International Journal of Forecasting, Elsevier, vol. 29(1), pages 13-27.
    14. Graham Elliott & Allan Timmermann, 2016. "Forecasting in Economics and Finance," Annual Review of Economics, Annual Reviews, vol. 8(1), pages 81-110, October.
    15. Artūras Juodis & Simas Kučinskas, 2023. "Quantifying noise in survey expectations," Quantitative Economics, Econometric Society, vol. 14(2), pages 609-650, May.
    16. Andrew Martinez, 2017. "Testing for Differences in Path Forecast Accuracy: Forecast-Error Dynamics Matter," Working Papers (Old Series) 1717, Federal Reserve Bank of Cleveland.
    17. Kontogeorgos, Georgios & Lambrias, Kyriacos, 2019. "An analysis of the Eurosystem/ECB projections," Working Paper Series 2291, European Central Bank.
    18. Jean-Yves Pitarakis, 2020. "A Novel Approach to Predictive Accuracy Testing in Nested Environments," Papers 2008.08387, arXiv.org, revised Oct 2023.
    19. Oguzhan Akgun & Alain Pirotte & Giovanni Urga & Zhenlin Yang, 2020. "Equal Predictive Ability Tests Based on Panel Data with Applications to OECD and IMF Forecasts," Papers 2003.02803, arXiv.org, revised Feb 2023.
    20. Daniel Borup & Martin Thyrsgaard, 2017. "Statistical tests for equal predictive ability across multiple forecasting methods," CREATES Research Papers 2017-19, Department of Economics and Business Economics, Aarhus University.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:econom:v:237:y:2023:i:2:s0304407621002256. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/jeconom .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.