IDEAS home Printed from https://ideas.repec.org/p/pra/mprapa/55017.html
   My bibliography  Save this paper

Theoretical guidelines for a partially informed forecast examiner

Author

Listed:
  • Tsyplakov, Alexander

Abstract

The paper explores probability theory foundations behind evaluation of probabilistic forecasts. The emphasis is on a situation when the forecast examiner possesses only partially the information which was available and was used to produce a forecast. We argue that in such a situation forecasts should be judged by their conditional auto-calibration. Necessary and sufficient conditions of auto-calibration are discussed and expressed in the form of testable moment conditions. The paper also analyzes relationships between forecast calibration and forecast efficiency.

Suggested Citation

  • Tsyplakov, Alexander, 2014. "Theoretical guidelines for a partially informed forecast examiner," MPRA Paper 55017, University Library of Munich, Germany.
  • Handle: RePEc:pra:mprapa:55017
    as

    Download full text from publisher

    File URL: https://mpra.ub.uni-muenchen.de/55017/1/MPRA_paper_55017.pdf
    File Function: original version
    Download Restriction: no

    File URL: https://mpra.ub.uni-muenchen.de/67333/1/MPRA_paper_67333.pdf
    File Function: revised version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Engelberg, Joseph & Manski, Charles F. & Williams, Jared, 2009. "Comparing the Point Predictions and Subjective Probability Distributions of Professional Forecasters," Journal of Business & Economic Statistics, American Statistical Association, vol. 27, pages 30-41.
    2. Berkowitz, Jeremy, 2001. "Testing Density Forecasts, with Applications to Risk Management," Journal of Business & Economic Statistics, American Statistical Association, vol. 19(4), pages 465-474, October.
    3. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    4. Galbraith, John W. & van Norden, Simon, 2011. "Kernel-based calibration diagnostics for recession and inflation probability forecasts," International Journal of Forecasting, Elsevier, vol. 27(4), pages 1041-1057, October.
    5. Paul H. Kupiec, 1995. "Techniques for verifying the accuracy of risk measurement models," Finance and Economics Discussion Series 95-24, Board of Governors of the Federal Reserve System (U.S.).
    6. Boero, Gianna & Smith, Jeremy & Wallis, Kenneth F., 2011. "Scoring rules and survey density forecasts," International Journal of Forecasting, Elsevier, vol. 27(2), pages 379-393.
    7. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    8. Michael P. Clements & Nick Taylor, 2003. "Evaluating interval forecasts of high-frequency financial data," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 18(4), pages 445-456.
    9. Hall, Stephen G. & Mitchell, James, 2007. "Combining density forecasts," International Journal of Forecasting, Elsevier, vol. 23(1), pages 1-13.
    10. Amisano, Gianni & Giacomini, Raffaella, 2007. "Comparing Density Forecasts via Weighted Likelihood Ratio Tests," Journal of Business & Economic Statistics, American Statistical Association, vol. 25, pages 177-190, April.
    11. Diebold, Francis X & Rudebusch, Glenn D, 1989. "Scoring the Leading Indicators," The Journal of Business, University of Chicago Press, vol. 62(3), pages 369-391, July.
    12. Giorgio Valente & Lucio Sarno, 2004. "Comparing the accuracy of density forecasts from competing models," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 23(8), pages 541-557.
    13. Christoffersen, Peter F, 1998. "Evaluating Interval Forecasts," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 841-862, November.
    14. Wallis, Kenneth F., 2003. "Chi-squared tests of interval and density forecasts, and the Bank of England's fan charts," International Journal of Forecasting, Elsevier, vol. 19(2), pages 165-175.
    15. Diks, Cees & Panchenko, Valentyn & van Dijk, Dick, 2011. "Likelihood-based scoring rules for comparing density forecasts in tails," Journal of Econometrics, Elsevier, vol. 163(2), pages 215-230, August.
    16. Michael P. Clements & David I. Harvey, 2010. "Forecast encompassing tests and probability forecasts," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(6), pages 1028-1062.
    17. Tilmann Gneiting & Fadoua Balabdaoui & Adrian E. Raftery, 2007. "Probabilistic forecasts, calibration and sharpness," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 69(2), pages 243-268, April.
    18. Yi‐Ting Chen, 2011. "Moment tests for density forecast evaluation in the presence of parameter estimation uncertainty," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 30(4), pages 409-450, July.
    19. James Mitchell & Kenneth F. Wallis, 2011. "Evaluating density forecasts: forecast combinations, model mixtures, calibration and sharpness," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 26(6), pages 1023-1040, September.
    20. Michael Clements, 2006. "Evaluating the survey of professional forecasters probability distributions of expected inflation based on derived event probability forecasts," Empirical Economics, Springer, vol. 31(1), pages 49-64, March.
    21. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    22. Tsyplakov, Alexander, 2011. "Evaluating density forecasts: a comment," MPRA Paper 31184, University Library of Munich, Germany.
    23. repec:hal:journl:peer-00834423 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tobias Fissler & Jana Hlavinov'a & Birgit Rudloff, 2019. "Elicitability and Identifiability of Systemic Risk Measures," Papers 1907.01306, arXiv.org, revised Oct 2019.
    2. Marc-Oliver Pohle, 2020. "The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation," Papers 2005.01835, arXiv.org.
    3. Natalia Nolde & Johanna F. Ziegel, 2016. "Elicitability and backtesting: Perspectives for banking regulation," Papers 1608.05498, arXiv.org, revised Feb 2017.
    4. Ziegel, Johanna F. & Krueger, Fabian & Jordan, Alexander & Fasciati, Fernando, 2017. "Murphy Diagrams: Forecast Evaluation of Expected Shortfall," Working Papers 0632, University of Heidelberg, Department of Economics.
    5. Taylor, James W., 2020. "A strategic predictive distribution for tests of probabilistic calibration," International Journal of Forecasting, Elsevier, vol. 36(4), pages 1380-1388.
    6. Werner Ehm & Tilmann Gneiting & Alexander Jordan & Fabian Krüger, 2016. "Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(3), pages 505-562, June.
    7. Johanna F. Ziegel & Fabian Kruger & Alexander Jordan & Fernando Fasciati, 2017. "Murphy Diagrams: Forecast Evaluation of Expected Shortfall," Papers 1705.04537, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Weron, Rafał, 2014. "Electricity price forecasting: A review of the state-of-the-art with a look into the future," International Journal of Forecasting, Elsevier, vol. 30(4), pages 1030-1081.
    2. Diks, Cees & Panchenko, Valentyn & van Dijk, Dick, 2011. "Likelihood-based scoring rules for comparing density forecasts in tails," Journal of Econometrics, Elsevier, vol. 163(2), pages 215-230, August.
    3. Tsyplakov, Alexander, 2013. "Evaluation of Probabilistic Forecasts: Proper Scoring Rules and Moments," MPRA Paper 45186, University Library of Munich, Germany.
    4. Gaglianone, Wagner Piazza & Marins, Jaqueline Terra Moura, 2017. "Evaluation of exchange rate point and density forecasts: An application to Brazil," International Journal of Forecasting, Elsevier, vol. 33(3), pages 707-728.
    5. Tsyplakov, Alexander, 2012. "Assessment of probabilistic forecasts: Proper scoring rules and moments," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 27(3), pages 115-132.
    6. Nowotarski, Jakub & Weron, Rafał, 2018. "Recent advances in electricity price forecasting: A review of probabilistic forecasting," Renewable and Sustainable Energy Reviews, Elsevier, vol. 81(P1), pages 1548-1568.
    7. Barbara Rossi, 2019. "Forecasting in the presence of instabilities: How do we know whether models predict well and how to improve them," Economics Working Papers 1711, Department of Economics and Business, Universitat Pompeu Fabra, revised Jul 2021.
    8. Cees Diks & Valentyn Panchenko & Dick van Dijk, 2008. "Partial Likelihood-Based Scoring Rules for Evaluating Density Forecasts in Tails," Tinbergen Institute Discussion Papers 08-050/4, Tinbergen Institute.
    9. Gordy, Michael B. & McNeil, Alexander J., 2020. "Spectral backtests of forecast distributions with application to risk management," Journal of Banking & Finance, Elsevier, vol. 116(C).
    10. repec:hal:journl:peer-00834423 is not listed on IDEAS
    11. Marc-Oliver Pohle, 2020. "The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation," Papers 2005.01835, arXiv.org.
    12. Wagner Piazza Gaglianone & Jaqueline Terra Moura Marins, 2014. "Risk Assessment of the Brazilian FX Rate," Working Papers Series 344, Central Bank of Brazil, Research Department.
    13. Lucio Sarno, 2003. "Nonlinear Exchange Rate Models: A Selective Overview," Rivista di Politica Economica, SIPI Spa, vol. 93(4), pages 3-46, July-Augu.
    14. Kolassa, Stephan, 2016. "Evaluating predictive count data distributions in retail sales forecasting," International Journal of Forecasting, Elsevier, vol. 32(3), pages 788-803.
    15. Clements, Michael P., 2018. "Are macroeconomic density forecasts informative?," International Journal of Forecasting, Elsevier, vol. 34(2), pages 181-198.
    16. Federico Bassetti & Roberto Casarin & Francesco Ravazzolo, 2019. "Density Forecasting," BEMPS - Bozen Economics & Management Paper Series BEMPS59, Faculty of Economics and Management at the Free University of Bozen.
    17. Diks, Cees & Fang, Hao, 2020. "Comparing density forecasts in a risk management context," International Journal of Forecasting, Elsevier, vol. 36(2), pages 531-551.
    18. esposito, francesco paolo & cummins, mark, 2015. "Multiple hypothesis testing of market risk forecasting models," MPRA Paper 64986, University Library of Munich, Germany.
    19. Taylor, James W., 2020. "A strategic predictive distribution for tests of probabilistic calibration," International Journal of Forecasting, Elsevier, vol. 36(4), pages 1380-1388.
    20. Anne Opschoor & Dick van Dijk & Michel van der Wel, 2014. "Improving Density Forecasts and Value-at-Risk Estimates by Combining Densities," Tinbergen Institute Discussion Papers 14-090/III, Tinbergen Institute.
    21. Borowska, Agnieszka & Hoogerheide, Lennart & Koopman, Siem Jan & van Dijk, Herman K., 2020. "Partially censored posterior for robust and efficient risk evaluation," Journal of Econometrics, Elsevier, vol. 217(2), pages 335-355.

    More about this item

    Keywords

    probabilistic forecast; forecast calibration; moment condition; probability integral transform; orthogonality condition; scoring rule; forecast encompassing;
    All these keywords.

    JEL classification:

    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pra:mprapa:55017. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/vfmunde.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Joachim Winter (email available below). General contact details of provider: https://edirc.repec.org/data/vfmunde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.