IDEAS home Printed from https://ideas.repec.org/p/bri/uobdis/15-661.html
   My bibliography  Save this paper

Counting Biased Forecasters: An Application of Multiple Testing Techniques

Author

Listed:
  • Fabiana Gomez
  • David Pacini

Abstract

We investigate the problem of counting biased forecasters among a group of unbiased and biased forecasters of macroeconomic variables. The innovation is to implement a procedure controlling for the expected proportion of unbiased forecasters that could be erroneously classified as biased (i.e., the false discovery rate). Monte Carlo exercises illustrate the relevance of controlling the false discovery rate in this context. Using data from the Survey of Professional Forecasters, we find that up to 7 out of 10 forecasters classified as biased by a procedure not controlling the false discovery rate may actually be unbiased.

Suggested Citation

  • Fabiana Gomez & David Pacini, 2015. "Counting Biased Forecasters: An Application of Multiple Testing Techniques," Bristol Economics Discussion Papers 15/661, School of Economics, University of Bristol, UK.
    • Handle: RePEc:bri:uobdis:15/661
    as

    Download full text from publisher

    File URL: http://www.bristol.ac.uk/efm/media/workingpapers/working_papers/pdffiles/dp15661.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Ivana Komunjer & Michael T. Owyang, 2012. "Multivariate Forecast Evaluation and Rationality Testing," The Review of Economics and Statistics, MIT Press, vol. 94(4), pages 1066-1080, November.
    2. Carlos Capistr¡N & Allan Timmermann, 2009. "Disagreement and Biases in Inflation Expectations," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 41(2-3), pages 365-396, March.
    3. Antonello D’Agostino & Kieran Mcquinn & Karl Whelan, 2012. "Are Some Forecasters Really Better Than Others?," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 44(4), pages 715-732, June.
    4. Batchelor, Roy & Dua, Pami, 1991. "Blue Chip Rationality Tests," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 23(4), pages 692-705, November.
    5. Keane, Michael P & Runkle, David E, 1990. "Testing the Rationality of Price Forecasts: New Evidence from Panel Data," American Economic Review, American Economic Association, vol. 80(4), pages 714-735, September.
    6. Brown, Bryan W & Maital, Shlomo, 1981. "What Do Economists Know? An Empirical Study of Experts' Expectations," Econometrica, Econometric Society, vol. 49(2), pages 491-504, March.
    7. James H. Stock & Mark W. Watson, 2007. "Why Has U.S. Inflation Become Harder to Forecast?," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 39(s1), pages 3-33, February.
    8. Croushore Dean, 2010. "An Evaluation of Inflation Forecasts from Surveys Using Real-Time Data," The B.E. Journal of Macroeconomics, De Gruyter, vol. 10(1), pages 1-32, May.
    9. Zarnowitz, Victor, 1985. "Rational Expectations and Macroeconomic Forecasts," Journal of Business & Economic Statistics, American Statistical Association, vol. 3(4), pages 293-311, October.
    10. Batchelor, R A, 1990. "All Forecasters Are Equal," Journal of Business & Economic Statistics, American Statistical Association, vol. 8(1), pages 143-144, January.
    11. Bonham, Carl S & Cohen, Richard H, 2001. "To Aggregate, Pool, or Neither: Testing the Rational-Expectations Hypothesis Using Survey Data," Journal of Business & Economic Statistics, American Statistical Association, vol. 19(3), pages 278-291, July.
    12. Jacob A. Mincer & Victor Zarnowitz, 1969. "The Evaluation of Economic Forecasts," NBER Chapters, in: Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance, pages 3-46, National Bureau of Economic Research, Inc.
    13. Davies, Anthony & Lahiri, Kajal, 1995. "A new framework for analyzing survey forecasts using three-dimensional panel data," Journal of Econometrics, Elsevier, vol. 68(1), pages 205-227, July.
    14. Scott Schuh, 2001. "An evaluation of recent macroeconomic forecast errors," New England Economic Review, Federal Reserve Bank of Boston, pages 35-56.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Michael P. Clements, 2022. "Forecaster Efficiency, Accuracy, and Disagreement: Evidence Using Individual‐Level Survey Data," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 54(2-3), pages 537-568, March.
    2. Croushore Dean, 2010. "An Evaluation of Inflation Forecasts from Surveys Using Real-Time Data," The B.E. Journal of Macroeconomics, De Gruyter, vol. 10(1), pages 1-32, May.
    3. Graham Elliott & Ivana Komunjer & Allan Timmermann, 2008. "Biases in Macroeconomic Forecasts: Irrationality or Asymmetric Loss?," Journal of the European Economic Association, MIT Press, vol. 6(1), pages 122-157, March.
    4. Fildes, Robert & Stekler, Herman, 2002. "The state of macroeconomic forecasting," Journal of Macroeconomics, Elsevier, vol. 24(4), pages 435-468, December.
    5. repec:lan:wpaper:470 is not listed on IDEAS
    6. Clements, Michael P., 2006. "Internal consistency of survey respondentsíforecasts: Evidence based on the Survey of Professional Forecasters," Economic Research Papers 269742, University of Warwick - Department of Economics.
    7. Clements, Michael P., 2021. "Do survey joiners and leavers differ from regular participants? The US SPF GDP growth and inflation forecasts," International Journal of Forecasting, Elsevier, vol. 37(2), pages 634-646.
    8. repec:spo:wpecon:info:hdl:2441/f4rshpf3v1umfa09lat09b1bg is not listed on IDEAS
    9. Carl Bonham & Richard Cohen & Shigeyuki Abe, 2006. "The Rationality and Heterogeneity of Survey Forecasts of the Yen-Dollar Exchange Rate: A Reexamination," Working Papers 200611, University of Hawaii at Manoa, Department of Economics.
    10. Silva Lopes, Artur, 1994. "A "hipótese das expectativas racionais": teoria e realidade (uma visita guiada à literatura até 1992) [The "rational expectations hypothesis": theory and reality (a guided tour ," MPRA Paper 9699, University Library of Munich, Germany, revised 23 Jul 2008.
    11. Julieta Caunedo & Riccardo Dicecio & Ivana Komunjer & Michael T. Owyang, 2020. "Asymmetry, Complementarities, and State Dependence in Federal Reserve Forecasts," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 52(1), pages 205-228, February.
    12. repec:hal:wpspec:info:hdl:2441/f4rshpf3v1umfa09lat09b1bg is not listed on IDEAS
    13. repec:lan:wpaper:425 is not listed on IDEAS
    14. repec:lan:wpaper:539557 is not listed on IDEAS
    15. repec:lan:wpaper:413 is not listed on IDEAS
    16. repec:hal:spmain:info:hdl:2441/f4rshpf3v1umfa09lat09b1bg is not listed on IDEAS
    17. Paul Hubert, 2010. "Monetary policy, imperfect information and the expectations channel [Politique monétaire,information imparfaite et canal des anticipations]," SciencePo Working papers Main tel-04095385, HAL.
    18. Paul Hubert, 2010. "Monetary Policy, Imperfect Information and the Expectations Channel," Sciences Po publications info:hdl:2441/f4rshpf3v1u, Sciences Po.
    19. Antonello D’Agostino & Kieran Mcquinn & Karl Whelan, 2012. "Are Some Forecasters Really Better Than Others?," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 44(4), pages 715-732, June.
    20. Isiklar, Gultekin, 2005. "On aggregation bias in fixed-event forecast efficiency tests," Economics Letters, Elsevier, vol. 89(3), pages 312-316, December.
    21. James M. Nason & Gregor W. Smith, 2021. "Measuring the slowly evolving trend in US inflation with professional forecasts," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 36(1), pages 1-17, January.
    22. Berge, Travis J., 2018. "Understanding survey-based inflation expectations," International Journal of Forecasting, Elsevier, vol. 34(4), pages 788-801.
    23. Capistrán, Carlos, 2008. "Bias in Federal Reserve inflation forecasts: Is the Federal Reserve irrational or just cautious?," Journal of Monetary Economics, Elsevier, vol. 55(8), pages 1415-1427, November.
    24. Ivana Komunjer & Michael T. Owyang, 2012. "Multivariate Forecast Evaluation and Rationality Testing," The Review of Economics and Statistics, MIT Press, vol. 94(4), pages 1066-1080, November.
    25. Geoff Kenny & Thomas Kostka & Federico Masera, 2015. "Can Macroeconomists Forecast Risk? Event-Based Evidence from the Euro-Area SPF," International Journal of Central Banking, International Journal of Central Banking, vol. 11(4), pages 1-46, December.
    26. Ager, P. & Kappler, M. & Osterloh, S., 2009. "The accuracy and efficiency of the Consensus Forecasts: A further application and extension of the pooled approach," International Journal of Forecasting, Elsevier, vol. 25(1), pages 167-181.
    27. Constantin Burgi, 2016. "What Do We Lose When We Average Expectations?," Working Papers 2016-013, The George Washington University, Department of Economics, H. O. Stekler Research Program on Forecasting.

    More about this item

    Keywords

    Biased Forecasters; Multiple Testing; False Discovery Rate.;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C23 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Models with Panel Data; Spatio-temporal Models
    • E17 - Macroeconomics and Monetary Economics - - General Aggregative Models - - - Forecasting and Simulation: Models and Applications

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bri:uobdis:15/661. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Vicky Jackson (email available below). General contact details of provider: https://edirc.repec.org/data/sebriuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.