IDEAS home Printed from https://ideas.repec.org/p/nya/albaec/13-02.html
   My bibliography  Save this paper

Testing the Value of Probability Forecasts for Calibrated Combining

Author

Listed:
  • Kajal Lahiri
  • Huaming Peng
  • Yongchen Zhao

Abstract

We combine the probability forecasts of real GDP declines from the U.S. Survey of Professional Forecasters, after trimming the forecasts that do not have "value" in the sense of Merton (1981). For this purpose, we propose a new test to evaluate probability forecasts that does not require converting the probabilities to binary forecasts before testing. The test accommodates serial correlation and skewness in the forecasts, and is implemented using a circular block bootstrap procedure. We find that the number of forecasters making valuable forecasts decreases sharply as horizon increases. The beta-transformed linear pool, based only on the valuable individual forecasts, is shown to outperform the simple average for all horizons on a number of performance measures including calibration and sharpness.

Suggested Citation

  • Kajal Lahiri & Huaming Peng & Yongchen Zhao, 2013. "Testing the Value of Probability Forecasts for Calibrated Combining," Discussion Papers 13-02, University at Albany, SUNY, Department of Economics.
  • Handle: RePEc:nya:albaec:13-02
    as

    Download full text from publisher

    File URL: http://www.albany.edu/economics/research/workingp/2013/lmz-test.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Pesaran, M. Hashem & Timmermann, Allan G., 1994. "A generalization of the non-parametric Henriksson-Merton test of market timing," Economics Letters, Elsevier, vol. 44(1-2), pages 1-7.
    2. Henriksson, Roy D & Merton, Robert C, 1981. "On Market Timing and Investment Performance. II. Statistical Procedures for Evaluating Forecasting Skills," The Journal of Business, University of Chicago Press, vol. 54(4), pages 513-533, October.
    3. Lahiri, Kajal & Monokroussos, George & Zhao, Yongchen, 2013. "The yield spread puzzle and the information content of SPF forecasts," Economics Letters, Elsevier, vol. 118(1), pages 219-221.
    4. Galbraith, John W. & van Norden, Simon, 2011. "Kernel-based calibration diagnostics for recession and inflation probability forecasts," International Journal of Forecasting, Elsevier, vol. 27(4), pages 1041-1057, October.
    5. Whitney K. Newey & Kenneth D. West, 1994. "Automatic Lag Selection in Covariance Matrix Estimation," Review of Economic Studies, Oxford University Press, vol. 61(4), pages 631-653.
    6. Guo, Jiin-Huarng & Luh, Wei-Ming, 2000. "An invertible transformation two-sample trimmed t-statistic under heterogeneity and nonnormality," Statistics & Probability Letters, Elsevier, vol. 49(1), pages 1-7, August.
    7. John W. Galbraith & Simon van Norden, 2012. "Assessing gross domestic product and inflation probability forecasts derived from Bank of England fan charts," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 175(3), pages 713-727, July.
    8. Pesaran, M Hashem & Timmermann, Allan, 1992. "A Simple Nonparametric Test of Predictive Performance," Journal of Business & Economic Statistics, American Statistical Association, vol. 10(4), pages 561-565, October.
    9. Clements, Michael P. & Harvey, David I., 2011. "Combining probability forecasts," International Journal of Forecasting, Elsevier, vol. 27(2), pages 208-223, April.
    10. Kenny, Geoff & Genre, Véronique & Meyler, Aidan & Timmermann, Allan, 2010. "Combining the forecasts in the ECB survey of professional forecasters: can anything beat the simple average?," Working Paper Series 1277, European Central Bank.
    11. Clements, Michael P., 2008. "Consensus and uncertainty: Using forecast probabilities of output declines," International Journal of Forecasting, Elsevier, vol. 24(1), pages 76-86.
    12. Tom Stark, 2010. "Realistic evaluation of real-time forecasts in the Survey of Professional Forecasters," Research Rap Special Report, Federal Reserve Bank of Philadelphia, issue May.
    13. James H. Stock & Mark W.Watson, 2003. "Forecasting Output and Inflation: The Role of Asset Prices," Journal of Economic Literature, American Economic Association, vol. 41(3), pages 788-829, September.
    14. Capistrán, Carlos & Timmermann, Allan, 2009. "Forecast Combination With Entry and Exit of Experts," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(4), pages 428-440.
    15. Michael P. Clements, 2011. "An Empirical Investigation of the Effects of Rounding on the SPF Probabilities of Decline and Output Growth Histograms," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 43(1), pages 207-220, February.
    16. Pesaran, M. Hashem & Timmermann, Allan, 2009. "Testing Dependence Among Serially Correlated Multicategory Variables," Journal of the American Statistical Association, American Statistical Association, vol. 104(485), pages 325-337.
    17. Roopesh Ranjan & Tilmann Gneiting, 2010. "Combining probability forecasts," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 72(1), pages 71-91, January.
    18. Michael P. Clements & David I. Harvey, 2010. "Forecast encompassing tests and probability forecasts," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(6), pages 1028-1062.
    19. Dean Croushore, 1993. "Introducing: the survey of professional forecasters," Business Review, Federal Reserve Bank of Philadelphia, issue Nov, pages 3-15.
    20. Schnader, M H & Stekler, H O, 1990. "Evaluating Predictions of Change," The Journal of Business, University of Chicago Press, vol. 63(1), pages 99-107, January.
    21. Newey, Whitney & West, Kenneth, 2014. "A simple, positive semi-definite, heteroscedasticity and autocorrelation consistent covariance matrix," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 33(1), pages 125-132.
    22. Lahiri, Kajal & Wang, J. George, 2013. "Evaluating probability forecasts for GDP declines using alternative methodologies," International Journal of Forecasting, Elsevier, vol. 29(1), pages 175-190.
    23. Rudebusch, Glenn D. & Williams, John C., 2009. "Forecasting Recessions: The Puzzle of the Enduring Power of the Yield Curve," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(4), pages 492-503.
    24. Merton, Robert C, 1981. "On Market Timing and Investment Performance. I. An Equilibrium Theory of Value for Market Forecasts," The Journal of Business, University of Chicago Press, vol. 54(3), pages 363-406, July.
    25. Aiolfi, Marco & Timmermann, Allan, 2006. "Persistence in forecasting performance and conditional combination strategies," Journal of Econometrics, Elsevier, vol. 135(1-2), pages 31-53.
    26. Kajal Lahiri & J George Wang, 2006. "Subjective Probability Forecasts for Recessions," Business Economics, Palgrave Macmillan;National Association for Business Economics, vol. 41(2), pages 26-37, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Constantin Bürgi & Tara M. Sinclair, 2017. "A nonparametric approach to identifying a subset of forecasters that outperforms the simple average," Empirical Economics, Springer, vol. 53(1), pages 101-115, August.
    2. Graham Elliott, 2017. "Forecast combination when outcomes are difficult to predict," Empirical Economics, Springer, vol. 53(1), pages 7-20, August.
    3. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    4. Yuri S. Popkov & Yuri A. Dubnov & Alexey Yu. Popkov, 2016. "New Method of Randomized Forecasting Using Entropy-Robust Estimation: Application to the World Population Prediction," Mathematics, MDPI, vol. 4(1), pages 1-16, March.
    5. Yongchen Zhao, 2020. "Predicting U.S. Business Cycle Turning Points Using Real-Time Diffusion Indexes Based on a Large Data Set," Journal of Business Cycle Research, Springer;Centre for International Research on Economic Tendency Surveys (CIRET), vol. 16(2), pages 77-97, November.
    6. Valentina Corradi & Sainan Jin & Norman R. Swanson, 2023. "Robust forecast superiority testing with an application to assessing pools of expert forecasters," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(4), pages 596-622, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lahiri, Kajal & Yang, Liu, 2013. "Forecasting Binary Outcomes," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1025-1106, Elsevier.
    2. Lahiri, Kajal & Wang, J. George, 2013. "Evaluating probability forecasts for GDP declines using alternative methodologies," International Journal of Forecasting, Elsevier, vol. 29(1), pages 175-190.
    3. Blaskowitz, Oliver & Herwartz, Helmut, 2014. "Testing the value of directional forecasts in the presence of serial correlation," International Journal of Forecasting, Elsevier, vol. 30(1), pages 30-42.
    4. Lahiri, Kajal & Yang, Liu, 2016. "Asymptotic variance of Brier (skill) score in the presence of serial correlation," Economics Letters, Elsevier, vol. 141(C), pages 125-129.
    5. Bespalova, Olga, 2018. "Forecast Evaluation in Macroeconomics and International Finance. Ph.D. thesis, George Washington University, Washington, DC, USA," MPRA Paper 117706, University Library of Munich, Germany.
    6. Lim, Terence & Lo, Andrew W. & Merton, Robert C. & Scholes, Myron S., 2006. "The Derivatives Sourcebook," Foundations and Trends(R) in Finance, now publishers, vol. 1(5–6), pages 365-572, April.
    7. Baghestani, Hamid & Toledo, Hugo, 2017. "Do analysts' forecasts of term spread differential help predict directional change in exchange rates?," International Review of Economics & Finance, Elsevier, vol. 47(C), pages 62-69.
    8. Oliver Blaskowitz & Helmut Herwartz, 2008. "Testing directional forecast value in the presence of serial correlation," SFB 649 Discussion Papers SFB649DP2008-073, Sonderforschungsbereich 649, Humboldt University, Berlin, Germany.
    9. IIZUKA Nobuo, 2013. "Predicting Business Cycle Phases by Professional Forecasters- Are They Useful ?," ESRI Discussion paper series 305, Economic and Social Research Institute (ESRI).
    10. Nyberg, Henri & Pönkä, Harri, 2016. "International sign predictability of stock returns: The role of the United States," Economic Modelling, Elsevier, vol. 58(C), pages 323-338.
    11. Tim Meyer, 2019. "On the Directional Accuracy of United States Housing Starts Forecasts: Evidence from Survey Data," The Journal of Real Estate Finance and Economics, Springer, vol. 58(3), pages 457-488, April.
    12. Mihaela Simionescu, 2014. "Directional accuracy for inflation and unemployment rate predictions in Romania," International Journal of Business and Economic Sciences Applied Research (IJBESAR), International Hellenic University (IHU), Kavala Campus, Greece (formerly Eastern Macedonia and Thrace Institute of Technology - EMaTTech), vol. 7(2), pages 129-138, September.
    13. Baghestani, Hamid, 2011. "Federal Reserve and private forecasts of growth in investment," Journal of Economics and Business, Elsevier, vol. 63(4), pages 290-305, July.
    14. Baris Soybilgen & Ege Yazgan, 2017. "An evaluation of inflation expectations in Turkey," Central Bank Review, Research and Monetary Policy Department, Central Bank of the Republic of Turkey, vol. 17(1), pages 1-31–38.
    15. Hamid Baghestani, 2010. "Evaluating Blue Chip forecasts of the trade-weighted dollar exchange rate," Applied Financial Economics, Taylor & Francis Journals, vol. 20(24), pages 1879-1889.
    16. McCracken,M.W. & West,K.D., 2001. "Inference about predictive ability," Working papers 14, Wisconsin Madison - Social Systems.
    17. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    18. Liu, Weiling & Moench, Emanuel, 2016. "What predicts US recessions?," International Journal of Forecasting, Elsevier, vol. 32(4), pages 1138-1150.
    19. Geoff Kenny & Thomas Kostka & Federico Masera, 2015. "Can Macroeconomists Forecast Risk? Event-Based Evidence from the Euro-Area SPF," International Journal of Central Banking, International Journal of Central Banking, vol. 11(4), pages 1-46, December.
    20. Ash, J. C. K. & Smyth, D. J. & Heravi, S. M., 1998. "Are OECD forecasts rational and useful?: a directional analysis," International Journal of Forecasting, Elsevier, vol. 14(3), pages 381-391, September.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nya:albaec:13-02. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Byoung Park (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.