IDEAS home Printed from https://ideas.repec.org/p/fip/fednsr/161.html
   My bibliography  Save this paper

Modeling uncertainty: predictive accuracy as a proxy for predictive confidence

Author

Listed:
  • Robert W. Rich
  • Joseph Tracy

Abstract

This paper evaluates current strategies for the empirical modeling of forecast behavior. In particular, we focus on the reliability of using proxies from time series models of heteroskedasticity to describe changes in predictive confidence. We address this issue by examining the relationship between ex post forecast errors and ex ante measures of forecast uncertainty from data on inflation forecasts from the Survey of Professional Forecasters. The results provide little evidence of a strong link between observed heteroskedasticity in the consensus forecast errors and forecast uncertainty. Instead, the findings indicate a significant link between observed heteroskedasticity in the consensus forecast errors and forecast dispersion. We conclude that conventional model-based measures of uncertainty may be capturing not the degree of confidence that individuals attach to their forecasts but rather the degree of disagreement across individuals in their forecasts.

Suggested Citation

  • Robert W. Rich & Joseph Tracy, 2003. "Modeling uncertainty: predictive accuracy as a proxy for predictive confidence," Staff Reports 161, Federal Reserve Bank of New York.
  • Handle: RePEc:fip:fednsr:161
    as

    Download full text from publisher

    File URL: https://www.newyorkfed.org/medialibrary/media/research/staff_reports/sr161.html
    Download Restriction: no

    File URL: https://www.newyorkfed.org/medialibrary/media/research/staff_reports/sr161.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Hansen, Lars Peter, 1982. "Large Sample Properties of Generalized Method of Moments Estimators," Econometrica, Econometric Society, vol. 50(4), pages 1029-1054, July.
    2. Lahiri, Kajal & Teigland, Christie & Zaporowski, Mark, 1988. "Interest Rates and the Subjective Probability Distribution of Inflation Forecasts," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 20(2), pages 233-248, May.
    3. Nicholas S. Souleles, 2001. "Consumer Sentiment: Its Rationality and Usefulness in Forecasting Expenditure - Evidence from the Michigan Micro Data," NBER Working Papers 8410, National Bureau of Economic Research, Inc.
    4. Batchelor, Roy & Dua, Pami, 1991. "Blue Chip Rationality Tests," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 23(4), pages 692-705, November.
    5. Victor Zarnowitz, 1984. "Business Cycles Analysis and Expectational Survey Data," NBER Working Papers 1378, National Bureau of Economic Research, Inc.
    6. Keane, Michael P & Runkle, David E, 1990. "Testing the Rationality of Price Forecasts: New Evidence from Panel Data," American Economic Review, American Economic Association, vol. 80(4), pages 714-735, September.
    7. Francis X. Diebold & Anthony S. Tay & Kenneth F. Wallis, 1997. "Evaluating Density Forecasts of Inflation: The Survey of Professional Forecasters," NBER Working Papers 6228, National Bureau of Economic Research, Inc.
    8. Bomberger, William A, 1996. "Disagreement as a Measure of Uncertainty," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 28(3), pages 381-392, August.
    9. Engle, Robert F. & White (the late), Halbert (ed.), 1999. "Cointegration, Causality, and Forecasting: Festschrift in Honour of Clive W. J. Granger," OUP Catalogue, Oxford University Press, number 9780198296836, Decembrie.
    10. Christopher D. Carroll, 2001. "The Epidemiology of Macroeconomic Expectations," NBER Working Papers 8695, National Bureau of Economic Research, Inc.
    11. Giordani, Paolo & Soderlind, Paul, 2003. "Inflation forecast uncertainty," European Economic Review, Elsevier, vol. 47(6), pages 1037-1059, December.
    12. Lloyd B. Thomas, 1999. "Survey Measures of Expected U.S. Inflation," Journal of Economic Perspectives, American Economic Association, vol. 13(4), pages 125-144, Fall.
    13. Bollerslev, Tim & Chou, Ray Y. & Kroner, Kenneth F., 1992. "ARCH modeling in finance : A review of the theory and empirical evidence," Journal of Econometrics, Elsevier, vol. 52(1-2), pages 5-59.
    14. Bonham, Carl & Cohen, Richard, 1995. "Testing the Rationality of Price Forecasts: Comment," American Economic Review, American Economic Association, vol. 85(1), pages 284-289, March.
    15. Engle, Robert F, 1982. "Autoregressive Conditional Heteroscedasticity with Estimates of the Variance of United Kingdom Inflation," Econometrica, Econometric Society, vol. 50(4), pages 987-1007, July.
    16. Cukierman, Alex & Wachtel, Paul, 1979. "Differential Inflationary Expectations and the Variability of the Rate of Inflation: Theory and Evidence," American Economic Review, American Economic Association, vol. 69(4), pages 595-609, September.
    17. Pennacchi, George G, 1991. "Identifying the Dynamics of Real Interest Rates and Inflation: Evidence Using Survey Data," Review of Financial Studies, Society for Financial Studies, vol. 4(1), pages 53-86.
    18. James H. Stock & Mark W. Watson, 1993. "Business Cycles, Indicators, and Forecasting," NBER Books, National Bureau of Economic Research, Inc, number stoc93-1, March.
    19. Lucas, Robert Jr., 1972. "Expectations and the neutrality of money," Journal of Economic Theory, Elsevier, vol. 4(2), pages 103-124, April.
    20. Dean Croushore, 1993. "Introducing: the survey of professional forecasters," Business Review, Federal Reserve Bank of Philadelphia, issue Nov, pages 3-15.
    21. Stock, James H. & Watson, Mark W. (ed.), 1993. "Business Cycles, Indicators, and Forecasting," National Bureau of Economic Research Books, University of Chicago Press, edition 1, number 9780226774886, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Hartmann, Matthias & Herwartz, Helmut & Ulm, Maren, 2017. "A comparative assessment of alternative ex ante measures of inflation uncertainty," International Journal of Forecasting, Elsevier, vol. 33(1), pages 76-89.
    2. Michael P. Clements, 2014. "US Inflation Expectations and Heterogeneous Loss Functions, 1968–2010," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 33(1), pages 1-14, January.
    3. Jonas Dovern & Ulrich Fritsche, 2008. "Estimating Fundamental Cross-Section Dispersion from Fixed Event Forecasts," Discussion Papers of DIW Berlin 787, DIW Berlin, German Institute for Economic Research.
    4. Hussain, Syed Mujahid, 2011. "Simultaneous monetary policy announcements and international stock markets response: An intraday analysis," Journal of Banking & Finance, Elsevier, vol. 35(3), pages 752-764, March.
    5. Van Nieuwerburgh, Stijn & Veldkamp, Laura, 2006. "Learning asymmetries in real business cycles," Journal of Monetary Economics, Elsevier, vol. 53(4), pages 753-772, May.
    6. Kajal Lahiri & Fushang Liu, 2006. "ARCH Models for Multi-period Forecast Uncertainty: A Reality Check Using a Panel of Density Forecasts," Advances in Econometrics, in: Econometric Analysis of Financial and Economic Time Series, pages 321-363, Emerald Group Publishing Limited.
    7. Capistrán, Carlos, 2008. "Bias in Federal Reserve inflation forecasts: Is the Federal Reserve irrational or just cautious?," Journal of Monetary Economics, Elsevier, vol. 55(8), pages 1415-1427, November.
    8. Andersen, Torben G. & Bollerslev, Tim & Diebold, Francis X. & Vega, Clara, 2007. "Real-time price discovery in global stock, bond and foreign exchange markets," Journal of International Economics, Elsevier, vol. 73(2), pages 251-277, November.
    9. Clements, Michael P., 2006. "Internal consistency of survey respondentsíforecasts: Evidence based on the Survey of Professional Forecasters," Economic Research Papers 269742, University of Warwick - Department of Economics.
    10. Fushang Liu & Kajal Lahiri, 2006. "Modelling multi-period inflation uncertainty using a panel of density forecasts," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 21(8), pages 1199-1219.
    11. Carlos Capistr¡N & Allan Timmermann, 2009. "Disagreement and Biases in Inflation Expectations," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 41(2-3), pages 365-396, March.
    12. Stefania D'Amico & Athanasios Orphanides, 2008. "Uncertainty and disagreement in economic forecasting," Finance and Economics Discussion Series 2008-56, Board of Governors of the Federal Reserve System (U.S.).
    13. Carlos Capistrán & Allan Timmermann, 2009. "Disagreement and Biases in Inflation Expectations," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 41(2‐3), pages 365-396, March.
    14. Clements, Michael P., 2008. "Consensus and uncertainty: Using forecast probabilities of output declines," International Journal of Forecasting, Elsevier, vol. 24(1), pages 76-86.
    15. Andersson, Magnus & Hansen, Lars Jul & Sebestyén, Szabolcs, 2006. "Which news moves the euro area bond market?," Working Paper Series 631, European Central Bank.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kajal Lahiri & Fushang Liu, 2006. "ARCH Models for Multi-period Forecast Uncertainty: A Reality Check Using a Panel of Density Forecasts," Advances in Econometrics, in: Econometric Analysis of Financial and Economic Time Series, pages 321-363, Emerald Group Publishing Limited.
    2. Clements, Michael P., 2010. "Explanations of the inconsistencies in survey respondents' forecasts," European Economic Review, Elsevier, vol. 54(4), pages 536-549, May.
    3. Davies, Antony, 2006. "A framework for decomposing shocks and measuring volatilities derived from multi-dimensional panel data of survey forecasts," International Journal of Forecasting, Elsevier, vol. 22(2), pages 373-393.
    4. Harvey, David I. & Newbold, Paul, 2003. "The non-normality of some macroeconomic forecast errors," International Journal of Forecasting, Elsevier, vol. 19(4), pages 635-653.
    5. Clements, Michael P., 2006. "Internal consistency of survey respondentsíforecasts: Evidence based on the Survey of Professional Forecasters," Economic Research Papers 269742, University of Warwick - Department of Economics.
    6. Kajal Lahiri & Fushang Liu, 2006. "Modelling multi‐period inflation uncertainty using a panel of density forecasts," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 21(8), pages 1199-1219, December.
    7. Giordani, Paolo & Soderlind, Paul, 2003. "Inflation forecast uncertainty," European Economic Review, Elsevier, vol. 47(6), pages 1037-1059, December.
    8. Carl S Bonham & Richard H Cohen, 2000. "Testing the Rational Expectations Hypothesis using Survey Data," Working Papers 200007, University of Hawaii at Manoa, Department of Economics.
    9. Ball, Laurence & Croushore, Dean, 2003. "Expectations and the Effects of Monetary Policy," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 35(4), pages 473-484, August.
    10. Francis X. Diebold, 1998. "The Past, Present, and Future of Macroeconomic Forecasting," Journal of Economic Perspectives, American Economic Association, vol. 12(2), pages 175-192, Spring.
    11. N. Gregory Mankiw & Ricardo Reis & Justin Wolfers, 2004. "Disagreement about Inflation Expectations," NBER Chapters, in: NBER Macroeconomics Annual 2003, Volume 18, pages 209-270, National Bureau of Economic Research, Inc.
    12. Clements, Michael P, 2012. "Subjective and Ex Post Forecast Uncertainty : US Inflation and Output Growth," The Warwick Economics Research Paper Series (TWERPS) 995, University of Warwick, Department of Economics.
    13. Clements, Michael P., 2014. "Probability distributions or point predictions? Survey forecasts of US output growth and inflation," International Journal of Forecasting, Elsevier, vol. 30(1), pages 99-117.
    14. Clements, Michael P., 2016. "Long-run restrictions and survey forecasts of output, consumption and investment," International Journal of Forecasting, Elsevier, vol. 32(3), pages 614-628.
    15. Clements, Michael P., 2008. "Rounding of probability forecasts : The SPF forecast probabilities of negative output growth," The Warwick Economics Research Paper Series (TWERPS) 869, University of Warwick, Department of Economics.
    16. Jean Sepulveda-Umanzor, 2004. "The Relation Between Macroeconomic Uncertainty And The Expected Performance Of the Economy," Econometric Society 2004 Latin American Meetings 304, Econometric Society.
    17. Kajal Lahiri & Xuguang Sheng, 2010. "Measuring forecast uncertainty by disagreement: The missing link," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 514-538.
    18. Clements, Michael P., 2008. "Consensus and uncertainty: Using forecast probabilities of output declines," International Journal of Forecasting, Elsevier, vol. 24(1), pages 76-86.
    19. Graham Elliott & Ivana Komunjer & Allan Timmermann, 2008. "Biases in Macroeconomic Forecasts: Irrationality or Asymmetric Loss?," Journal of the European Economic Association, MIT Press, vol. 6(1), pages 122-157, March.
    20. Engelberg, Joseph & Manski, Charles F. & Williams, Jared, 2009. "Comparing the Point Predictions and Subjective Probability Distributions of Professional Forecasters," Journal of Business & Economic Statistics, American Statistical Association, vol. 27, pages 30-41.

    More about this item

    Keywords

    Forecasting; time series analysis; Business cycles;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fip:fednsr:161. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Gabriella Bucciarelli (email available below). General contact details of provider: https://edirc.repec.org/data/frbnyus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.