IDEAS home Printed from https://ideas.repec.org/p/fip/fednrp/9525.html
   My bibliography  Save this paper

Forecast evaluation and combination

Author

Listed:
  • Francis X. Diebold
  • Jose A. Lopez

Abstract

Forecasts are of great importance and widely used in economics and finance. Quite simply, good forecasts lead to good decisions. The importance of forecast evaluation and combination techniques follows immediatelyforecast users naturally have a keen interest in monitoring and improving forecast performance. Here we provide a five-part selective account of forecast evaluation and combination methods. In the first, we discuss evaluation of a single forecast, and in particular, evaluation of whether and how it may be improved. In the second, we discuss the evaluation and comparison of the accuracy of competing forecasts. In the third, we discuss whether and how a set of forecasts may be combined to produce a superior composite forecast. In the fourth, we describe a number of forecast evaluation topics of particular relevance in economics and finance, including methods for evaluating direction-of-change forecasts, probability forecasts and volatility forecasts. In the fifth, we conclude.

Suggested Citation

  • Francis X. Diebold & Jose A. Lopez, 1995. "Forecast evaluation and combination," Research Paper 9525, Federal Reserve Bank of New York.
  • Handle: RePEc:fip:fednrp:9525
    as

    Download full text from publisher

    File URL: https://www.newyorkfed.org/medialibrary/media/research/staff_reports/research_papers/9525.pdf
    Download Restriction: no

    File URL: https://www.newyorkfed.org/medialibrary/media/research/staff_reports/research_papers/9525.html
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Francis X. Diebold & Lee E. Ohanian & Jeremy Berkowitz, 1998. "Dynamic Equilibrium Economies: A Framework for Comparing Models and Data," Review of Economic Studies, Oxford University Press, vol. 65(3), pages 433-451.
    2. Fair, Ray C & Shiller, Robert J, 1989. "The Informational Context of Ex Ante Forecasts," The Review of Economics and Statistics, MIT Press, vol. 71(2), pages 325-331, May.
    3. Brown, Bryan W & Maital, Shlomo, 1981. "What Do Economists Know? An Empirical Study of Experts' Expectations," Econometrica, Econometric Society, vol. 49(2), pages 491-504, March.
    4. Hendry, David F & Mizon, Grayham E, 1978. "Serial Correlation as a Convenient Simplification, not a Nuisance: A Comment on a Study of the Demand for Money by the Bank of England," Economic Journal, Royal Economic Society, vol. 88(351), pages 549-563, September.
    5. Hansen, Lars Peter & Hodrick, Robert J, 1980. "Forward Exchange Rates as Optimal Predictors of Future Spot Rates: An Econometric Analysis," Journal of Political Economy, University of Chicago Press, vol. 88(5), pages 829-853, October.
    6. Fair, Ray C & Shiller, Robert J, 1990. "Comparing Information in Forecasts from Econometric Models," American Economic Review, American Economic Association, vol. 80(3), pages 375-389, June.
    7. M. H. Pesaran, 1974. "On the General Problem of Model Selection," Review of Economic Studies, Oxford University Press, vol. 41(2), pages 153-171.
    8. Cumby, Robert E. & Modest, David M., 1987. "Testing for market timing ability : A framework for forecast evaluation," Journal of Financial Economics, Elsevier, vol. 19(1), pages 169-189, September.
    9. Christoffersen, Peter F & Diebold, Francis X, 1996. "Further Results on Forecasting and Model Selection under Asymmetric Loss," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 11(5), pages 561-571, Sept.-Oct.
    10. Shiller, Robert J, 1979. "The Volatility of Long-Term Interest Rates and Expectations Models of the Term Structure," Journal of Political Economy, University of Chicago Press, vol. 87(6), pages 1190-1219, December.
    11. John Y. Campbell & N. Gregory Mankiw, 1987. "Are Output Fluctuations Transitory?," The Quarterly Journal of Economics, Oxford University Press, vol. 102(4), pages 857-880.
    12. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    13. Clements, M.P. & Hendry, D., 1992. "On the Limitations of Comparing Mean Square Forecast Errors," Economics Series Working Papers 99138, University of Oxford, Department of Economics.
    14. Dufour, J.M., 1979. "Rank Tests for Serial Dependence," Cahiers de recherche 7815, Universite de Montreal, Departement de sciences economiques.
    15. Clemen, Robert T., 1989. "Combining forecasts: A review and annotated bibliography," International Journal of Forecasting, Elsevier, vol. 5(4), pages 559-583.
    16. Bollerslev, Tim & Ghysels, Eric, 1996. "Periodic Autoregressive Conditional Heteroscedasticity," Journal of Business & Economic Statistics, American Statistical Association, vol. 14(2), pages 139-151, April.
    17. Pagan, Adrian R. & Schwert, G. William, 1990. "Alternative models for conditional stock volatility," Journal of Econometrics, Elsevier, vol. 45(1-2), pages 267-290.
    18. Pesaran, M Hashem & Timmermann, Allan, 1992. "A Simple Nonparametric Test of Predictive Performance," Journal of Business & Economic Statistics, American Statistical Association, vol. 10(4), pages 561-565, October.
    19. Coulson, N.E. & Robins, R.P., 1989. "Forecast Combination In A Dynamic Setting," Papers 8-88-4, Pennsylvania State - Department of Economics.
    20. West, Kenneth D. & Edison, Hali J. & Cho, Dongchul, 1993. "A utility-based comparison of some models of exchange rate volatility," Journal of International Economics, Elsevier, vol. 35(1-2), pages 23-45, August.
    21. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    22. Schnader, M H & Stekler, H O, 1990. "Evaluating Predictions of Change," The Journal of Business, University of Chicago Press, vol. 63(1), pages 99-107, January.
    23. Fama, Eugene F & French, Kenneth R, 1988. "Permanent and Temporary Components of Stock Prices," Journal of Political Economy, University of Chicago Press, vol. 96(2), pages 246-273, April.
    24. Cumby, Robert E & Huizinga, John, 1992. "Testing the Autocorrelation Structure of Disturbances in Ordinary Least Squares and Instrumental Variables Regressions," Econometrica, Econometric Society, vol. 60(1), pages 185-195, January.
    25. Bollerslev, Tim & Engle, Robert F. & Nelson, Daniel B., 1986. "Arch models," Handbook of Econometrics, in: R. F. Engle & D. McFadden (ed.), Handbook of Econometrics, edition 1, volume 4, chapter 49, pages 2959-3038, Elsevier.
    26. Yock Y. Chong & David F. Hendry, 1986. "Econometric Evaluation of Linear Macro-Economic Models," Review of Economic Studies, Oxford University Press, vol. 53(4), pages 671-690.
    27. Robert F. Engle & Che-Hsiung Hong & Alex Kane, 1990. "Valuation of Variance Forecast with Simulated Option Markets," NBER Working Papers 3350, National Bureau of Economic Research, Inc.
    28. McCulloch, Robert & Rossi, Peter E., 1990. "Posterior, predictive, and utility-based approaches to testing the arbitrage pricing theory," Journal of Financial Economics, Elsevier, vol. 28(1-2), pages 7-38.
    29. Meese, Richard A & Rogoff, Kenneth, 1988. " Was It Real? The Exchange Rate-Interest Differential Relation over the Modern Floating-Rate Period," Journal of Finance, American Finance Association, vol. 43(4), pages 933-948, September.
    30. Fama, Eugene F, 1975. "Short-Term Interest Rates as Predictors of Inflation," American Economic Review, American Economic Association, vol. 65(3), pages 269-282, June.
    31. Campbell, Bryan & Dufour, Jean-Marie, 1995. "Exact Nonparametric Orthogonality and Random Walk Tests," The Review of Economics and Statistics, MIT Press, vol. 77(1), pages 1-16, February.
    32. Christoffersen, Peter F. & Diebold, Francis X., 1997. "Optimal Prediction Under Asymmetric Loss," Econometric Theory, Cambridge University Press, vol. 13(6), pages 808-817, December.
    33. Campbell, Bryan & Ghysels, Eric, 1995. "Federal Budget Projections: A Nonparametric Assessment of Bias and Efficiency," The Review of Economics and Statistics, MIT Press, vol. 77(1), pages 17-31, February.
    34. Cochrane, John H, 1988. "How Big Is the Random Walk in GNP?," Journal of Political Economy, University of Chicago Press, vol. 96(5), pages 893-920, October.
    35. Granger, C. W. J. & Newbold, Paul, 1986. "Forecasting Economic Time Series," Elsevier Monographs, Elsevier, edition 2, number 9780122951831 edited by Shell, Karl, December.
    36. Fama, Eugene F, 1970. "Efficient Capital Markets: A Review of Theory and Empirical Work," Journal of Finance, American Finance Association, vol. 25(2), pages 383-417, May.
    37. Diebold, Francis X. & Pauly, Peter, 1990. "The use of prior information in forecast combination," International Journal of Forecasting, Elsevier, vol. 6(4), pages 503-508, December.
    38. Engle, Robert F & Kozicki, Sharon, 1993. "Testing for Common Features," Journal of Business & Economic Statistics, American Statistical Association, vol. 11(4), pages 369-380, October.
    39. Nelson, Charles R & Schwert, G William, 1977. "Short-Term Interest Rates as Predictors of Inflation: On Testing the Hypothesis That the Real Rate of Interest is Constant," American Economic Review, American Economic Association, vol. 67(3), pages 478-486, June.
    40. Murphy, Allan H. & Winkler, Robert L., 1992. "Diagnostic verification of probability forecasts," International Journal of Forecasting, Elsevier, vol. 7(4), pages 435-455, March.
    41. Diebold, Francis X, 1988. "Serial Correlation and the Combination of Forecasts," Journal of Business & Economic Statistics, American Statistical Association, vol. 6(1), pages 105-111, January.
    42. Merton, Robert C, 1981. "On Market Timing and Investment Performance. I. An Equilibrium Theory of Value for Market Forecasts," The Journal of Business, University of Chicago Press, vol. 54(3), pages 363-406, July.
    43. Diebold, Francis X & Rudebusch, Glenn D, 1989. "Scoring the Leading Indicators," The Journal of Business, University of Chicago Press, vol. 62(3), pages 369-391, July.
    44. Francis X. Diebold & Peter Pauly, 1986. "Structural change and the combination of forecasts," Special Studies Papers 201, Board of Governors of the Federal Reserve System (U.S.).
    45. Lopez, Jose A, 2001. "Evaluating the Predictive Accuracy of Volatility Models," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 20(2), pages 87-109, March.
    46. Bonham, Carl & Cohen, Richard, 1995. "Testing the Rationality of Price Forecasts: Comment," American Economic Review, American Economic Association, vol. 85(1), pages 284-289, March.
    47. Christoffersen, Peter F, 1998. "Evaluating Interval Forecasts," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 841-862, November.
    48. Diebold, Francis X. & Lindner, Peter, 1996. "Fractional integration and interval prediction," Economics Letters, Elsevier, vol. 50(3), pages 305-313, March.
    49. Campbell, Bryan & Dufour, Jean-Marie, 1991. "Over-rejections in rational expectations models : A non-parametric approach to the Mankiw-Shapiro problem," Economics Letters, Elsevier, vol. 35(3), pages 285-290, March.
    50. Leitch, Gordon & Tanner, J Ernest, 1991. "Economic Forecast Evaluation: Profits versus the Conventional Error Measures," American Economic Review, American Economic Association, vol. 81(3), pages 580-590, June.
    51. Engle, Robert F & Kozicki, Sharon, 1993. "Testing for Common Features: Reply," Journal of Business & Economic Statistics, American Statistical Association, vol. 11(4), pages 393-395, October.
    52. LeRoy, Stephen F & Porter, Richard D, 1981. "The Present-Value Relation: Tests Based on Implied Variance Bounds," Econometrica, Econometric Society, vol. 49(3), pages 555-574, May.
    53. Deutsch, Melinda & Granger, Clive W. J. & Terasvirta, Timo, 1994. "The combination of forecasts using changing weights," International Journal of Forecasting, Elsevier, vol. 10(1), pages 47-57, June.
    54. Mark, Nelson C, 1995. "Exchange Rates and Fundamentals: Evidence on Long-Horizon Predictability," American Economic Review, American Economic Association, vol. 85(1), pages 201-218, March.
    55. Clemon, Robert T & Winkler, Robert L, 1986. "Combining Economic Forecasts," Journal of Business & Economic Statistics, American Statistical Association, vol. 4(1), pages 39-46, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. McCracken,M.W. & West,K.D., 2001. "Inference about predictive ability," Working papers 14, Wisconsin Madison - Social Systems.
    2. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    3. Franses,Philip Hans & Dijk,Dick van, 2000. "Non-Linear Time Series Models in Empirical Finance," Cambridge Books, Cambridge University Press, number 9780521770415.
    4. Lahiri, Kajal & Yang, Liu, 2013. "Forecasting Binary Outcomes," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1025-1106, Elsevier.
    5. LeBaron, Blake, 2003. "Non-Linear Time Series Models in Empirical Finance,: Philip Hans Franses and Dick van Dijk, Cambridge University Press, Cambridge, 2000, 296 pp., Paperback, ISBN 0-521-77965-0, $33, [UK pound]22.95, [," International Journal of Forecasting, Elsevier, vol. 19(4), pages 751-752.
    6. Elkin Castaño & Luis Fernando Melo, 1998. "Métodos de Combinación de Pronósticos: Una Aplicación a la Inflación Colombiana," Borradores de Economia 109, Banco de la Republica de Colombia.
    7. Peter F. Christoffersen & Francis X. Diebold, 2006. "Financial Asset Returns, Direction-of-Change Forecasting, and Volatility Dynamics," Management Science, INFORMS, vol. 52(8), pages 1273-1287, August.
    8. Elliott, Graham & Komunjer, Ivana & Timmermann, Allan G, 2003. "Estimating Loss Function Parameters," CEPR Discussion Papers 3821, C.E.P.R. Discussion Papers.
    9. repec:zbw:cfswop:wp200408 is not listed on IDEAS
    10. McCracken, Michael W., 2007. "Asymptotics for out of sample tests of Granger causality," Journal of Econometrics, Elsevier, vol. 140(2), pages 719-752, October.
    11. Elkin Castaño Vélez & Luis Fernando Melo Velandia, 2000. "Metodos de combinacion de pronosticos: una aplicacion a la inflacion," Lecturas de Economía, Universidad de Antioquia, Departamento de Economía, issue 52, pages 113-165, Enero Jun.
    12. Lopez, Jose A, 2001. "Evaluating the Predictive Accuracy of Volatility Models," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 20(2), pages 87-109, March.
    13. Graham Elliott & Ivana Komunjer & Allan Timmermann, 2008. "Biases in Macroeconomic Forecasts: Irrationality or Asymmetric Loss?," Journal of the European Economic Association, MIT Press, vol. 6(1), pages 122-157, March.
    14. Graham Elliott & Allan Timmermann, 2016. "Economic Forecasting," Economics Books, Princeton University Press, edition 1, number 10740.
    15. Francis X. Diebold & Todd A. Gunther & Anthony S. Tay, "undated". "Evaluating Density Forecasts," CARESS Working Papres 97-18, University of Pennsylvania Center for Analytic Research and Economics in the Social Sciences.
    16. West, Kenneth D., 2001. "Encompassing tests when no model is encompassing," Journal of Econometrics, Elsevier, vol. 105(1), pages 287-308, November.
    17. West, Kenneth D., 2006. "Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 3, pages 99-134, Elsevier.
    18. Clements, Michael P. & Harvey, David I., 2011. "Combining probability forecasts," International Journal of Forecasting, Elsevier, vol. 27(2), pages 208-223.
    19. Peter Christoffersen & Francis X. Diebold, 2002. "Financial Asset Returns, Market Timing, and Volatility Dynamics," CIRANO Working Papers 2002s-02, CIRANO.
    20. Timmermann, Allan, 2006. "Forecast Combinations," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 4, pages 135-196, Elsevier.
    21. Gregory Connor & Lisa R. Goldberg & Robert A. Korajczyk, 2010. "Portfolio Risk Analysis," Economics Books, Princeton University Press, edition 1, number 9224.

    More about this item

    Keywords

    Forecasting;

    JEL classification:

    • C5 - Mathematical and Quantitative Methods - - Econometric Modeling

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fip:fednrp:9525. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/frbnyus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (email available below). General contact details of provider: https://edirc.repec.org/data/frbnyus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.