IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v27yi2p379-393.html
   My bibliography  Save this article

Scoring rules and survey density forecasts

Author

Listed:
  • Boero, Gianna
  • Smith, Jeremy
  • Wallis, Kenneth F.

Abstract

This article provides a practical evaluation of some leading density forecast scoring rules in the context of forecast surveys. We analyse the density forecasts of UK inflation obtained from the Bank of England's Survey of External Forecasters, considering both the survey average forecasts published in the Bank's quarterly Inflation Report, and the individual survey responses recently made available to researchers by the Bank. The density forecasts are collected in histogram format, and the ranked probability score (RPS) is shown to have clear advantages over other scoring rules. Missing observations are a feature of forecast surveys, and we introduce an adjustment to the RPS, based on the Yates decomposition, to improve its comparative measurement of forecaster performance in the face of differential non-response. The new measure, denoted RPS*, is recommended to analysts of forecast surveys.

Suggested Citation

  • Boero, Gianna & Smith, Jeremy & Wallis, Kenneth F., 2011. "Scoring rules and survey density forecasts," International Journal of Forecasting, Elsevier, vol. 27(2), pages 379-393, April.
    • Handle: RePEc:eee:intfor:v:27:y::i:2:p:379-393
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0169-2070(10)00096-8
    Download Restriction: Full text for ScienceDirect subscribers only
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    2. Engelberg, Joseph & Manski, Charles F. & Williams, Jared, 2009. "Comparing the Point Predictions and Subjective Probability Distributions of Professional Forecasters," Journal of Business & Economic Statistics, American Statistical Association, vol. 27, pages 30-41.
    3. Gianna Boero & Jeremy Smith & KennethF. Wallis, 2008. "Uncertainty and Disagreement in Economic Prediction: The Bank of England Survey of External Forecasters," Economic Journal, Royal Economic Society, vol. 118(530), pages 1107-1127, July.
    4. Gianna Boero & Jeremy Smith & Kenneth F. Wallis, 2008. "Uncertainty and Disagreement in Economic Prediction: The Bank of England Survey of External Forecasters," Economic Journal, Royal Economic Society, vol. 118(530), pages 1107-1127, July.
    5. Boero, Gianna & Smith, Jeremy & Wallis, Kenneth F., 2008. "Here is the news: forecast revisions in the Bank of England survey of external forecasters," National Institute Economic Review, National Institute of Economic and Social Research, vol. 203, pages 68-77, January.
    6. P. Frevert, 1971. "Note," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 38(2), pages 269-270.
    7. Victor Zarnowitz, 1969. "The New ASA–NBER Survey of Forecasts by Economic Statisticians," NBER Chapters, in: Supplement to NBER Report Four, pages 1-8, National Bureau of Economic Research, Inc.
    8. Yates, J. Frank, 1988. "Analyzing the accuracy of probability judgments for multiple events: An extension of the covariance decomposition," Organizational Behavior and Human Decision Processes, Elsevier, vol. 41(3), pages 281-299, June.
    9. Amisano, Gianni & Giacomini, Raffaella, 2007. "Comparing Density Forecasts via Weighted Likelihood Ratio Tests," Journal of Business & Economic Statistics, American Statistical Association, vol. 25, pages 177-190, April.
    10. Groen, Jan J.J. & Kapetanios, George & Price, Simon, 2009. "A real time evaluation of Bank of England forecasts of inflation and growth," International Journal of Forecasting, Elsevier, vol. 25(1), pages 74-80.
    11. Boero, Gianna & Smith, Jeremy & Wallis, Kenneth F., 2008. "Evaluating a three-dimensional panel of point forecasts: The Bank of England Survey of External Forecasters," International Journal of Forecasting, Elsevier, vol. 24(3), pages 354-367.
    12. Capistrán, Carlos & Timmermann, Allan, 2009. "Forecast Combination With Entry and Exit of Experts," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(4), pages 428-440.
    13. Casillas-Olvera, Gabriel & Bessler, David A., 2006. "Probability forecasting and central bank accountability," Journal of Policy Modeling, Elsevier, vol. 28(2), pages 223-234, February.
    14. John W. Galbraith & Simon van Norden, 2009. "Calibration and Resolution Diagnostics for Bank of England Density Forecasts," CIRANO Working Papers 2009s-36, CIRANO.
    15. Kenneth F. Wallis, 2004. "An Assessment of Bank of England and National Institute Inflation Forecast Uncertainties," National Institute Economic Review, National Institute of Economic and Social Research, vol. 189(1), pages 64-71, July.
    16. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Krüger, Fabian & Nolte, Ingmar, 2016. "Disagreement versus uncertainty: Evidence from distribution forecasts," Journal of Banking & Finance, Elsevier, vol. 72(S), pages 172-186.
    2. Clements, Michael P, 2012. "Subjective and Ex Post Forecast Uncertainty : US Inflation and Output Growth," The Warwick Economics Research Paper Series (TWERPS) 995, University of Warwick, Department of Economics.
    3. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    4. Knüppel, Malte & Schultefrankenfeld, Guido, 2019. "Assessing the uncertainty in central banks’ inflation outlooks," International Journal of Forecasting, Elsevier, vol. 35(4), pages 1748-1769.
    5. Galvão, Ana Beatriz & Garratt, Anthony & Mitchell, James, 2021. "Does judgment improve macroeconomic density forecasts?," International Journal of Forecasting, Elsevier, vol. 37(3), pages 1247-1260.
    6. Zanetti Chini, Emilio, 2018. "Forecasting dynamically asymmetric fluctuations of the U.S. business cycle," International Journal of Forecasting, Elsevier, vol. 34(4), pages 711-732.
    7. Fabian Krüger & Ingmar Nolte, 2011. "Disagreement, Uncertainty and the True Predictive Density," Working Paper Series of the Department of Economics, University of Konstanz 2011-43, Department of Economics, University of Konstanz.
    8. Pfajfar, D. & Zakelj, B., 2012. "Uncertainty and Disagreement in Forecasting Inflation : Evidence from the Laboratory (Revised version of CentER DP 2011-053)," Other publications TiSEM 38fac5ce-fe8f-4b61-a679-f, Tilburg University, School of Economics and Management.
    9. Glas, Alexander, 2020. "Five dimensions of the uncertainty–disagreement linkage," International Journal of Forecasting, Elsevier, vol. 36(2), pages 607-627.
    10. Schultefrankenfeld Guido, 2013. "Forecast uncertainty and the Bank of England’s interest rate decisions," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 17(1), pages 1-20, February.
    11. Victor Lopez-Perez, 2016. "Macroeconomic Forecast Uncertainty In The Euro Area," Equilibrium. Quarterly Journal of Economics and Economic Policy, Institute of Economic Research, vol. 11(1), pages 9-41, March.
    12. Malte Knüppel & Guido Schultefrankenfeld, 2017. "Interest rate assumptions and predictive accuracy of central bank forecasts," Empirical Economics, Springer, vol. 53(1), pages 195-215, August.
    13. Tsuchiya, Yoichi, 2022. "Evaluating the European Central Bank’s uncertainty forecasts," Economic Analysis and Policy, Elsevier, vol. 73(C), pages 321-330.
    14. Gaglianone, Wagner Piazza & Marins, Jaqueline Terra Moura, 2017. "Evaluation of exchange rate point and density forecasts: An application to Brazil," International Journal of Forecasting, Elsevier, vol. 33(3), pages 707-728.
    15. Pfajfar, D. & Zakelj, B., 2012. "Uncertainty and Disagreement in Forecasting Inflation : Evidence from the Laboratory (Revised version of EBC DP 2011-014)," Other publications TiSEM 2b92a09f-918e-4614-978d-0, Tilburg University, School of Economics and Management.
    16. Monica Billio & Roberto Casarin & Francesco Ravazzolo & Herman K. van Dijk, 2011. "Combining Predictive Densities using Nonlinear Filtering with Applications to US Economics Data," Tinbergen Institute Discussion Papers 11-172/4, Tinbergen Institute.
    17. Barbara Rossi, 2019. "Forecasting in the presence of instabilities: How do we know whether models predict well and how to improve them," Economics Working Papers 1711, Department of Economics and Business, Universitat Pompeu Fabra, revised Jul 2021.
    18. Yanwei Jia & Jussi Keppo & Ville Satopää, 2023. "Herding in Probabilistic Forecasts," Management Science, INFORMS, vol. 69(5), pages 2713-2732, May.
    19. Kajal Lahiri & Xuguang Sheng, 2010. "Measuring forecast uncertainty by disagreement: The missing link," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 514-538.
    20. Billio, Monica & Casarin, Roberto & Ravazzolo, Francesco & van Dijk, Herman K., 2013. "Time-varying combinations of predictive densities using nonlinear filtering," Journal of Econometrics, Elsevier, vol. 177(2), pages 213-232.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:27:y::i:2:p:379-393. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.