IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v27yi2p379-393.html
   My bibliography  Save this article

Scoring rules and survey density forecasts

Author

Listed:
  • Boero, Gianna
  • Smith, Jeremy
  • Wallis, Kenneth F.

Abstract

This article provides a practical evaluation of some leading density forecast scoring rules in the context of forecast surveys. We analyse the density forecasts of UK inflation obtained from the Bank of England's Survey of External Forecasters, considering both the survey average forecasts published in the Bank's quarterly Inflation Report, and the individual survey responses recently made available to researchers by the Bank. The density forecasts are collected in histogram format, and the ranked probability score (RPS) is shown to have clear advantages over other scoring rules. Missing observations are a feature of forecast surveys, and we introduce an adjustment to the RPS, based on the Yates decomposition, to improve its comparative measurement of forecaster performance in the face of differential non-response. The new measure, denoted RPS*, is recommended to analysts of forecast surveys.

Suggested Citation

  • Boero, Gianna & Smith, Jeremy & Wallis, Kenneth F., 2011. "Scoring rules and survey density forecasts," International Journal of Forecasting, Elsevier, vol. 27(2), pages 379-393, April.
  • Handle: RePEc:eee:intfor:v:27:y::i:2:p:379-393
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0169-2070(10)00096-8
    Download Restriction: Full text for ScienceDirect subscribers only

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. John Galbraith & Simon van Norden, 2009. "Calibration and Resolution Diagnostics for Bank of England Density Forecasts," CIRANO Working Papers 2009s-36, CIRANO.
    2. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    3. Kenneth F. Wallis, 2004. "An Assessment of Bank of England and National Institute Inflation Forecast Uncertainties," National Institute Economic Review, National Institute of Economic and Social Research, vol. 189(1), pages 64-71, July.
    4. Engelberg, Joseph & Manski, Charles F. & Williams, Jared, 2009. "Comparing the Point Predictions and Subjective Probability Distributions of Professional Forecasters," Journal of Business & Economic Statistics, American Statistical Association, vol. 27, pages 30-41.
    5. Gianna Boero & Jeremy Smith & KennethF. Wallis, 2008. "Uncertainty and Disagreement in Economic Prediction: The Bank of England Survey of External Forecasters," Economic Journal, Royal Economic Society, vol. 118(530), pages 1107-1127, July.
    6. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    7. Capistrán, Carlos & Timmermann, Allan, 2009. "Forecast Combination With Entry and Exit of Experts," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(4), pages 428-440.
    8. Gianna Boero & Jeremy Smith & Kenneth F. Wallis, 2008. "Here is the News: Forecast Revisions in the Bank of England Survey of External Forecasters," National Institute Economic Review, National Institute of Economic and Social Research, vol. 203(1), pages 68-77, January.
    9. P. Frevert, 1971. "Note," Review of Economic Studies, Oxford University Press, vol. 38(2), pages 269-270.
    10. Victor Zarnowitz, 1969. "The New ASA–NBER Survey of Forecasts by Economic Statisticians," NBER Chapters,in: Supplement to NBER Report Four, pages 1-8 National Bureau of Economic Research, Inc.
    11. Yates, J. Frank, 1988. "Analyzing the accuracy of probability judgments for multiple events: An extension of the covariance decomposition," Organizational Behavior and Human Decision Processes, Elsevier, vol. 41(3), pages 281-299, June.
    12. Amisano, Gianni & Giacomini, Raffaella, 2007. "Comparing Density Forecasts via Weighted Likelihood Ratio Tests," Journal of Business & Economic Statistics, American Statistical Association, vol. 25, pages 177-190, April.
    13. Groen, Jan J.J. & Kapetanios, George & Price, Simon, 2009. "A real time evaluation of Bank of England forecasts of inflation and growth," International Journal of Forecasting, Elsevier, vol. 25(1), pages 74-80.
    14. Boero, Gianna & Smith, Jeremy & Wallis, Kenneth F., 2008. "Evaluating a three-dimensional panel of point forecasts: The Bank of England Survey of External Forecasters," International Journal of Forecasting, Elsevier, vol. 24(3), pages 354-367.
    15. Casillas-Olvera, Gabriel & Bessler, David A., 2006. "Probability forecasting and central bank accountability," Journal of Policy Modeling, Elsevier, vol. 28(2), pages 223-234, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Clements, Michael P., 2018. "Are macroeconomic density forecasts informative?," International Journal of Forecasting, Elsevier, vol. 34(2), pages 181-198.
    2. Ng, Jason & Forbes, Catherine S. & Martin, Gael M. & McCabe, Brendan P.M., 2013. "Non-parametric estimation of forecast distributions in non-Gaussian, non-linear state space models," International Journal of Forecasting, Elsevier, vol. 29(3), pages 411-430.
    3. Berg, Tim O. & Henzel, Steffen R., 2015. "Point and density forecasts for the euro area using Bayesian VARs," International Journal of Forecasting, Elsevier, vol. 31(4), pages 1067-1095.
    4. Ambrocio, Gene, 2017. "The real effects of overconfidence and fundamental uncertainty shocks," Research Discussion Papers 37, Bank of Finland.
    5. Laurent L. Pauwels & Andrey L. Vasnev, 2017. "Forecast combination for discrete choice models: predicting FOMC monetary policy decisions," Empirical Economics, Springer, vol. 52(1), pages 229-254, February.
    6. BAN Kanemi & KAWAGOE Masaaki & MATSUOKA Hideaki, 2013. "Evaluating Density Forecasts with Applications to ESPF," ESRI Discussion paper series 302, Economic and Social Research Institute (ESRI).
    7. Jan-Egbert Sturm & Jakob Haan, 2011. "Does central bank communication really lead to better forecasts of policy decisions? New evidence based on a Taylor rule model for the ECB," Review of World Economics (Weltwirtschaftliches Archiv), Springer;Institut für Weltwirtschaft (Kiel Institute for the World Economy), vol. 147(1), pages 41-58, April.
    8. Ambrocio, Gene, 2017. "The real effects of overconfidence and fundamental uncertainty shocks," Research Discussion Papers 37/2017, Bank of Finland.
    9. Geoff Kenny & Thomas Kostka & Federico Masera, 2015. "Density characteristics and density forecast performance: a panel analysis," Empirical Economics, Springer, vol. 48(3), pages 1203-1231, May.
    10. Corona Francisco & Wiper Michael Peter & Horrillo Juan de Dios Tena, 2017. "On the importance of the probabilistic model in identifying the most decisive games in a tournament," Journal of Quantitative Analysis in Sports, De Gruyter, vol. 13(1), pages 11-23, March.
    11. Rosa, Carlo, 2011. "Words that shake traders," Journal of Empirical Finance, Elsevier, vol. 18(5), pages 915-934.
    12. Tsyplakov, Alexander, 2014. "Theoretical guidelines for a partially informed forecast examiner," MPRA Paper 55017, University Library of Munich, Germany.
    13. David Harris & Gael M. Martin & Indeewara Perera & Don S. Poskitt, 2017. "Construction and visualization of optimal confidence sets for frequentist distributional forecasts," Monash Econometrics and Business Statistics Working Papers 9/17, Monash University, Department of Econometrics and Business Statistics.
    14. repec:syb:wpbsba:01/2013 is not listed on IDEAS

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:27:y::i:2:p:379-393. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Dana Niculescu). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.