IDEAS home Printed from https://ideas.repec.org/p/wpa/wuwpgt/0412002.html
   My bibliography  Save this paper

Correspondence On the Selection of Error Measures for Comparisons Among Forecasting Methods

Author

Listed:
  • JS Armstrong

    (The Wharton School - University of Pennsylvania)

  • Robert Fildes

    (The Management School - Lancaster University - UK)

Abstract

Clements and Hendry (1993) proposed the Generalized Forecast Error Second Moment (GFESM) as an improvement to the Mean Square Error in comparing forecasting performance across data series. They based their conclusion on the fact that rankings based on GFESM remain unaltered if the series are linearly transformed. In this paper, we argue that this evaluation ignores other important criteria. Also, their conclusions were illustrated by a simulation study whose relationship to real data was not obvious. Thirdly, prior empirical studies show that the mean square error is an inappropriate measure to serve as a basis for comparison. This undermines the claims made for the GFESM.

Suggested Citation

  • JS Armstrong & Robert Fildes, 2004. "Correspondence On the Selection of Error Measures for Comparisons Among Forecasting Methods," General Economics and Teaching 0412002, University Library of Munich, Germany.
    • Handle: RePEc:wpa:wuwpgt:0412002
    Note: Type of Document - pdf; pages: 4
    as

    Download full text from publisher

    File URL: https://econwpa.ub.uni-muenchen.de/econ-wp/get/papers/0412/0412002.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Makridakis, Spyros & Hibon, Michele & Lusk, Ed & Belhadjali, Moncef, 1987. "Confidence intervals: An empirical investigation of the series in the M-competition," International Journal of Forecasting, Elsevier, vol. 3(3-4), pages 489-508.
    2. Zellner, Arnold, 1986. "A tale of forecasting 1001 series : The Bayesian knight strikes again," International Journal of Forecasting, Elsevier, vol. 2(4), pages 491-494.
    3. Clements, M.P. & Hendry, D., 1992. "On the Limitations of Comparing Mean Square Forecast Errors," Economics Series Working Papers 99138, University of Oxford, Department of Economics.
    4. Robert Carbone & JS Armstrong, 2004. "Evaluation of Extrapolative Forecasting Methods: Results of a Survey of Academicians and Practitioners," General Economics and Teaching 0412008, University Library of Munich, Germany.
    5. Murphy, Allan H. & Winkler, Robert L., 1992. "Diagnostic verification of probability forecasts," International Journal of Forecasting, Elsevier, vol. 7(4), pages 435-455, March.
    6. Armstrong, J. Scott & Collopy, Fred, 1992. "Error measures for generalizing about forecasting methods: Empirical comparisons," International Journal of Forecasting, Elsevier, vol. 8(1), pages 69-80, June.
    7. Thompson, Patrick A., 1990. "An MSE statistic for comparing forecast accuracy across series," International Journal of Forecasting, Elsevier, vol. 6(2), pages 219-227, July.
    8. Price, D. H. R. & Sharp, J. A., 1986. "A comparison of the performance of different univariate forecasting methods in a model of capacity acquisition in UK electricity supply," International Journal of Forecasting, Elsevier, vol. 2(3), pages 333-348.
    9. Fildes, Robert, 1992. "The evaluation of extrapolative forecasting methods," International Journal of Forecasting, Elsevier, vol. 8(1), pages 81-98, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wagatha, Matthias, 2007. "Integration, Kointegration und die Langzeitprognose von Kreditausfallzyklen [Integration, Cointegration and Long-Horizont Forecasting of Credit-Default-Cycles]," MPRA Paper 8602, University Library of Munich, Germany.
    2. Hyndman, Rob J. & Koehler, Anne B., 2006. "Another look at measures of forecast accuracy," International Journal of Forecasting, Elsevier, vol. 22(4), pages 679-688.
    3. Jan G. De Gooijer & Rob J. Hyndman, 2005. "25 Years of IIF Time Series Forecasting: A Selective Review," Monash Econometrics and Business Statistics Working Papers 12/05, Monash University, Department of Econometrics and Business Statistics.
    4. De Gooijer, Jan G. & Hyndman, Rob J., 2006. "25 years of time series forecasting," International Journal of Forecasting, Elsevier, vol. 22(3), pages 443-473.
    5. Everette S. Gardner, 1999. "Note: Rule-Based Forecasting vs. Damped-Trend Exponential Smoothing," Management Science, INFORMS, vol. 45(8), pages 1169-1176, August.
    6. repec:lan:wpaper:413 is not listed on IDEAS
    7. repec:lan:wpaper:470 is not listed on IDEAS
    8. Shah, Chandra, 1997. "Model selection in univariate time series forecasting using discriminant analysis," International Journal of Forecasting, Elsevier, vol. 13(4), pages 489-500, December.
    9. Thury, Gerhard & Witt, Stephen F., 1998. "Forecasting industrial production using structural time series models," Omega, Elsevier, vol. 26(6), pages 751-767, December.
    10. Davydenko, Andrey & Fildes, Robert, 2013. "Measuring forecasting accuracy: The case of judgmental adjustments to SKU-level demand forecasts," International Journal of Forecasting, Elsevier, vol. 29(3), pages 510-522.
    11. repec:lan:wpaper:425 is not listed on IDEAS
    12. repec:lan:wpaper:539557 is not listed on IDEAS
    13. Garcia-Ferrer, Antonio & Bujosa-Brun, Marcos, 2000. "Forecasting OECD industrial turning points using unobserved components models with business survey data," International Journal of Forecasting, Elsevier, vol. 16(2), pages 207-227.
    14. Kumar, V. & Leone, Robert P. & Gaskins, John N., 1995. "Aggregate and disaggregate sector forecasting using consumer confidence measures," International Journal of Forecasting, Elsevier, vol. 11(3), pages 361-377, September.
    15. Garcia-Ferrer, Antonio & Queralt, Ricardo A., 1997. "A note on forecasting international tourism demand in Spain," International Journal of Forecasting, Elsevier, vol. 13(4), pages 539-549, December.
    16. Tashman, Leonard J. & Kruk, Joshua M., 1996. "The use of protocols to select exponential smoothing procedures: A reconsideration of forecasting competitions," International Journal of Forecasting, Elsevier, vol. 12(2), pages 235-253, June.
    17. Bruce G. S. Hardie & Peter S. Fader & Robert Zeithammer, 2003. "Forecasting new product trial in a controlled test market environment," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 22(5), pages 391-410.
    18. Kim, Sungil & Kim, Heeyoung, 2016. "A new metric of absolute percentage error for intermittent demand forecasts," International Journal of Forecasting, Elsevier, vol. 32(3), pages 669-679.
    19. Francis X. Diebold & Jose A. Lopez, 1995. "Forecast evaluation and combination," Research Paper 9525, Federal Reserve Bank of New York.
    20. Armstrong, J. Scott & Brodie, Roderick J., 1999. "Forecasting for Marketing," MPRA Paper 81690, University Library of Munich, Germany.
    21. Fred Collopy & J. Scott Armstrong, 1992. "Rule-Based Forecasting: Development and Validation of an Expert Systems Approach to Combining Time Series Extrapolations," Management Science, INFORMS, vol. 38(10), pages 1394-1414, October.
    22. Gary Madden & Joachim Tan, 2008. "Forecasting international bandwidth capacity using linear and ANN methods," Applied Economics, Taylor & Francis Journals, vol. 40(14), pages 1775-1787.
    23. David F. Hendry, 2002. "Forecast Failure, Expectations Formation and the Lucas Critique," Annals of Economics and Statistics, GENES, issue 67-68, pages 21-40.
    24. Darbellay, Georges A. & Slama, Marek, 2000. "Forecasting the short-term demand for electricity: Do neural networks stand a better chance?," International Journal of Forecasting, Elsevier, vol. 16(1), pages 71-83.

    More about this item

    Keywords

    ;

    JEL classification:

    • A - General Economics and Teaching

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wpa:wuwpgt:0412002. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: EconWPA The email address of this maintainer does not seem to be valid anymore. Please ask EconWPA to update the entry or send us the correct address (email available below). General contact details of provider: https://econwpa.ub.uni-muenchen.de .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.