IDEAS home Printed from https://ideas.repec.org/p/pra/mprapa/107403.html
   My bibliography  Save this paper

The Mean Squared Prediction Error Paradox

Author

Listed:
  • Pincheira, Pablo
  • Hardy, Nicolas

Abstract

In this paper, we show that traditional comparisons of Mean Squared Prediction Error (MSPE) between two competing forecasts may be highly controversial. This is so because when some specific conditions of efficiency are not met, the forecast displaying the lowest MSPE will also display the lowest correlation with the target variable. Given that violations of efficiency are usual in the forecasting literature, this opposite behavior in terms of accuracy and correlation with the target variable may be a fairly common empirical finding that we label here as "the MSPE Paradox." We characterize "Paradox zones" in terms of differences in correlation with the target variable and conduct some simple simulations to show that these zones may be non-empty sets. Finally, we illustrate the relevance of the Paradox with two empirical applications.

Suggested Citation

  • Pincheira, Pablo & Hardy, Nicolas, 2021. "The Mean Squared Prediction Error Paradox," MPRA Paper 107403, University Library of Munich, Germany.
    • Handle: RePEc:pra:mprapa:107403
    as

    Download full text from publisher

    File URL: https://mpra.ub.uni-muenchen.de/107403/1/MPRA_paper_107403.pdf
    File Function: original version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Ivo Welch & Amit Goyal, 2008. "A Comprehensive Look at The Empirical Performance of Equity Premium Prediction," The Review of Financial Studies, Society for Financial Studies, vol. 21(4), pages 1455-1508, July.
    2. Charles Engel & Kenneth D. West, 2005. "Exchange Rates and Fundamentals," Journal of Political Economy, University of Chicago Press, vol. 113(3), pages 485-517, June.
    3. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    4. Andrew Patton & Allan Timmermann, 2012. "Forecast Rationality Tests Based on Multi-Horizon Bounds," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(1), pages 1-17.
    5. Yu-Chin Chen & Kenneth S. Rogoff & Barbara Rossi, 2010. "Can Exchange Rates Forecast Commodity Prices?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(3), pages 1145-1194.
    6. Yash P. Mehra, 2002. "Survey measures of expected inflation : revisiting the issues of predictive content and rationality," Economic Quarterly, Federal Reserve Bank of Richmond, issue Sum, pages 17-36.
    7. Pablo Pincheira & Roberto Álvarez, 2009. "Evaluation of Short Run Inflation Forecasts and Forecasters in Chile," Money Affairs, CEMLA, vol. 0(2), pages 159-180, July-Dece.
    8. Clark, Todd E. & West, Kenneth D., 2007. "Approximately normal tests for equal predictive accuracy in nested models," Journal of Econometrics, Elsevier, vol. 138(1), pages 291-311, May.
    9. Bentancor, Andrea & Pincheira, Pablo, 2010. "Predicción de errores de proyección de inflación en Chile," El Trimestre Económico, Fondo de Cultura Económica, vol. 0(305), pages 129-154, enero-mar.
    10. Ang, Andrew & Bekaert, Geert & Wei, Min, 2007. "Do macro variables, asset markets, or surveys forecast inflation better?," Journal of Monetary Economics, Elsevier, vol. 54(4), pages 1163-1212, May.
    11. Domenico Ferraro & Kenneth S. Rogoff & Barbara Rossi, 2011. "Can oil prices forecast exchange rates?," Working Papers 11-34, Federal Reserve Bank of Philadelphia.
    12. Pincheira, Pablo & Hardy, Nicolás, 2021. "Forecasting aluminum prices with commodity currencies," Resources Policy, Elsevier, vol. 73(C).
    13. Clark, Todd E. & West, Kenneth D., 2006. "Using out-of-sample mean squared prediction errors to test the martingale difference hypothesis," Journal of Econometrics, Elsevier, vol. 135(1-2), pages 155-186.
    14. Souleles, Nicholas S, 2004. "Expectations, Heterogeneous Forecast Errors, and Consumption: Micro Evidence from the Michigan Consumer Sentiment Surveys," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 36(1), pages 39-72, February.
    15. Meese, Richard A & Rogoff, Kenneth, 1988. " Was It Real? The Exchange Rate-Interest Differential Relation over the Modern Floating-Rate Period," Journal of Finance, American Finance Association, vol. 43(4), pages 933-948, September.
    16. Ince, Onur & Molodtsova, Tanya, 2017. "Rationality and forecasting accuracy of exchange rate expectations: Evidence from survey-based forecasts," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 47(C), pages 131-151.
    17. Joutz, Fred & Stekler, H. O., 2000. "An evaluation of the predictions of the Federal Reserve," International Journal of Forecasting, Elsevier, vol. 16(1), pages 17-38.
    18. Pablo Pincheira, 2010. "A Real Time Evaluation of the Central Bank of Chile GDP Growth Forecasts," Money Affairs, CEMLA, vol. 0(1), pages 37-73, January-J.
    19. Ferraro, Domenico & Rogoff, Kenneth & Rossi, Barbara, 2015. "Can oil prices forecast exchange rates? An empirical analysis of the relationship between commodity prices and exchange rates," Journal of International Money and Finance, Elsevier, vol. 54(C), pages 116-141.
    20. Pablo Matías Pincheira Brown & Nicolás FernándeZ, 2011. "Corrección de algunos errores sistemáticos de predicción de inflación," Monetaria, CEMLA, vol. 0(1), pages 37-61, enero-mar.
    21. Jacob A. Mincer & Victor Zarnowitz, 1969. "The Evaluation of Economic Forecasts," NBER Chapters, in: Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance, pages 3-46, National Bureau of Economic Research, Inc.
    22. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    23. Pincheira Brown, Pablo & Hardy, Nicolás, 2019. "Forecasting base metal prices with the Chilean exchange rate," Resources Policy, Elsevier, vol. 62(C), pages 256-281.
    24. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    25. Timmermann, Allan, 2008. "Reply to the discussion of Elusive Return Predictability," International Journal of Forecasting, Elsevier, vol. 24(1), pages 29-30.
    26. Jacob A. Mincer, 1969. "Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance," NBER Books, National Bureau of Economic Research, Inc, number minc69-1, March.
    27. James H. Stock & Mark W.Watson, 2003. "Forecasting Output and Inflation: The Role of Asset Prices," Journal of Economic Literature, American Economic Association, vol. 41(3), pages 788-829, September.
    28. Nordhaus, William D, 1987. "Forecasting Efficiency: Concepts and Applications," The Review of Economics and Statistics, MIT Press, vol. 69(4), pages 667-674, November.
    29. Meese, Richard A. & Rogoff, Kenneth, 1983. "Empirical exchange rate models of the seventies : Do they fit out of sample?," Journal of International Economics, Elsevier, vol. 14(1-2), pages 3-24, February.
    30. Pablo Pincheira, 2012. "A Joint Test of Superior Predictive Ability for Chilean Inflation Forecasts," Journal Economía Chilena (The Chilean Economy), Central Bank of Chile, vol. 15(3), pages 04-39, December.
    31. Timmermann, Allan, 2008. "Elusive return predictability," International Journal of Forecasting, Elsevier, vol. 24(1), pages 1-18.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Nicolás Magner & Nicolás Hardy, 2022. "Cryptocurrency Forecasting: More Evidence of the Meese-Rogoff Puzzle," Mathematics, MDPI, vol. 10(13), pages 1-27, July.
    2. Pablo Pincheira & Nicolas Hardy & Andrea Bentancor, 2022. "A Simple Out-of-Sample Test of Predictability against the Random Walk Benchmark," Mathematics, MDPI, vol. 10(2), pages 1-20, January.
    3. Shafiullah Qureshi & Ba Chu & Fanny S. Demers, 2021. "Forecasting Canadian GDP Growth with Machine Learning," Carleton Economic Papers 21-05, Carleton University, Department of Economics.
    4. Pincheira-Brown, Pablo & Bentancor, Andrea & Hardy, Nicolás & Jarsun, Nabil, 2022. "Forecasting fuel prices with the Chilean exchange rate: Going beyond the commodity currency hypothesis," Energy Economics, Elsevier, vol. 106(C).
    5. Pincheira, Pablo & Hardy, Nicolas, 2022. "Correlation Based Tests of Predictability," MPRA Paper 112014, University Library of Munich, Germany.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Pincheira, Pablo & Hardy, Nicolas, 2020. "The Mean Squared Prediction Error Paradox: A summary," MPRA Paper 105020, University Library of Munich, Germany.
    2. Pincheira, Pablo & Hardy, Nicolas, 2022. "Correlation Based Tests of Predictability," MPRA Paper 112014, University Library of Munich, Germany.
    3. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    4. Nicolás Magner & Nicolás Hardy, 2022. "Cryptocurrency Forecasting: More Evidence of the Meese-Rogoff Puzzle," Mathematics, MDPI, vol. 10(13), pages 1-27, July.
    5. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    6. Barbara Rossi, 2019. "Forecasting in the presence of instabilities: How do we know whether models predict well and how to improve them," Economics Working Papers 1711, Department of Economics and Business, Universitat Pompeu Fabra, revised Jul 2021.
    7. Pablo Pincheira & Nicolás Hardy & Felipe Muñoz, 2021. "“Go Wild for a While!”: A New Test for Forecast Evaluation in Nested Models," Mathematics, MDPI, vol. 9(18), pages 1-28, September.
    8. Pincheira-Brown, Pablo & Neumann, Federico, 2020. "Can we beat the Random Walk? The case of survey-based exchange rate forecasts in Chile," Finance Research Letters, Elsevier, vol. 37(C).
    9. Barbara Rossi, 2013. "Exchange Rate Predictability," Journal of Economic Literature, American Economic Association, vol. 51(4), pages 1063-1119, December.
    10. Ahmed, Shamim & Liu, Xiaoquan & Valente, Giorgio, 2016. "Can currency-based risk factors help forecast exchange rates?," International Journal of Forecasting, Elsevier, vol. 32(1), pages 75-97.
    11. Feng, Wenjun & Zhang, Zhengjun, 2023. "Currency exchange rate predictability: The new power of Bitcoin prices," Journal of International Money and Finance, Elsevier, vol. 132(C).
    12. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    13. Ferraro, Domenico & Rogoff, Kenneth & Rossi, Barbara, 2015. "Can oil prices forecast exchange rates? An empirical analysis of the relationship between commodity prices and exchange rates," Journal of International Money and Finance, Elsevier, vol. 54(C), pages 116-141.
    14. Odendahl, Florens & Rossi, Barbara & Sekhposyan, Tatevik, 2023. "Evaluating forecast performance with state dependence," Journal of Econometrics, Elsevier, vol. 237(2).
    15. Raffaella Giacomini & Barbara Rossi, 2013. "Forecasting in macroeconomics," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 17, pages 381-408, Edward Elgar Publishing.
    16. Pincheira Brown, Pablo & Hardy, Nicolás, 2019. "Forecasting base metal prices with the Chilean exchange rate," Resources Policy, Elsevier, vol. 62(C), pages 256-281.
    17. Jörg Breitung & Malte Knüppel, 2021. "How far can we forecast? Statistical tests of the predictive content," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 36(4), pages 369-392, June.
    18. Granziera, Eleonora & Hubrich, Kirstin & Moon, Hyungsik Roger, 2014. "A predictability test for a small number of nested models," Journal of Econometrics, Elsevier, vol. 182(1), pages 174-185.
    19. Pincheira, Pablo & Hardy, Nicolás & Muñoz, Felipe, 2021. ""Go wild for a while!": A new asymptotically Normal test for forecast evaluation in nested models," MPRA Paper 105368, University Library of Munich, Germany.
    20. Rossi Junior, Jose Luiz & Felicio, Wilson Rafael de Oliveira, 2014. "Common Factors and the Exchange Rate: Results From the Brazilian Case," Revista Brasileira de Economia - RBE, EPGE Brazilian School of Economics and Finance - FGV EPGE (Brazil), vol. 68(1), April.

    More about this item

    Keywords

    Mean Squared Prediction Error; Correlation; Forecasting; Time Series; Random Walk.;
    All these keywords.

    JEL classification:

    • C1 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General
    • C10 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - General
    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • C2 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables
    • C22 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes
    • C4 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics
    • C40 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - General
    • C5 - Mathematical and Quantitative Methods - - Econometric Modeling
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods
    • C58 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Financial Econometrics
    • E0 - Macroeconomics and Monetary Economics - - General
    • E00 - Macroeconomics and Monetary Economics - - General - - - General
    • E30 - Macroeconomics and Monetary Economics - - Prices, Business Fluctuations, and Cycles - - - General (includes Measurement and Data)
    • E31 - Macroeconomics and Monetary Economics - - Prices, Business Fluctuations, and Cycles - - - Price Level; Inflation; Deflation
    • E37 - Macroeconomics and Monetary Economics - - Prices, Business Fluctuations, and Cycles - - - Forecasting and Simulation: Models and Applications
    • E44 - Macroeconomics and Monetary Economics - - Money and Interest Rates - - - Financial Markets and the Macroeconomy
    • E47 - Macroeconomics and Monetary Economics - - Money and Interest Rates - - - Forecasting and Simulation: Models and Applications
    • E52 - Macroeconomics and Monetary Economics - - Monetary Policy, Central Banking, and the Supply of Money and Credit - - - Monetary Policy
    • E58 - Macroeconomics and Monetary Economics - - Monetary Policy, Central Banking, and the Supply of Money and Credit - - - Central Banks and Their Policies
    • F30 - International Economics - - International Finance - - - General
    • F31 - International Economics - - International Finance - - - Foreign Exchange
    • F37 - International Economics - - International Finance - - - International Finance Forecasting and Simulation: Models and Applications
    • G00 - Financial Economics - - General - - - General
    • G12 - Financial Economics - - General Financial Markets - - - Asset Pricing; Trading Volume; Bond Interest Rates
    • G15 - Financial Economics - - General Financial Markets - - - International Financial Markets
    • G17 - Financial Economics - - General Financial Markets - - - Financial Forecasting and Simulation
    • Q0 - Agricultural and Natural Resource Economics; Environmental and Ecological Economics - - General
    • Q00 - Agricultural and Natural Resource Economics; Environmental and Ecological Economics - - General - - - General
    • Q02 - Agricultural and Natural Resource Economics; Environmental and Ecological Economics - - General - - - Commodity Market
    • Q1 - Agricultural and Natural Resource Economics; Environmental and Ecological Economics - - Agriculture
    • Q2 - Agricultural and Natural Resource Economics; Environmental and Ecological Economics - - Renewable Resources and Conservation
    • Q3 - Agricultural and Natural Resource Economics; Environmental and Ecological Economics - - Nonrenewable Resources and Conservation
    • Q33 - Agricultural and Natural Resource Economics; Environmental and Ecological Economics - - Nonrenewable Resources and Conservation - - - Resource Booms (Dutch Disease)
    • Q4 - Agricultural and Natural Resource Economics; Environmental and Ecological Economics - - Energy
    • Q43 - Agricultural and Natural Resource Economics; Environmental and Ecological Economics - - Energy - - - Energy and the Macroeconomy
    • Q47 - Agricultural and Natural Resource Economics; Environmental and Ecological Economics - - Energy - - - Energy Forecasting

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pra:mprapa:107403. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Joachim Winter (email available below). General contact details of provider: https://edirc.repec.org/data/vfmunde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.