IDEAS home Printed from https://ideas.repec.org/p/wpa/wuwpgt/0412002.html
   My bibliography  Save this paper

Correspondence On the Selection of Error Measures for Comparisons Among Forecasting Methods

Author

Listed:
  • JS Armstrong

    (The Wharton School - University of Pennsylvania)

  • Robert Fildes

    (The Management School - Lancaster University - UK)

Abstract

Clements and Hendry (1993) proposed the Generalized Forecast Error Second Moment (GFESM) as an improvement to the Mean Square Error in comparing forecasting performance across data series. They based their conclusion on the fact that rankings based on GFESM remain unaltered if the series are linearly transformed. In this paper, we argue that this evaluation ignores other important criteria. Also, their conclusions were illustrated by a simulation study whose relationship to real data was not obvious. Thirdly, prior empirical studies show that the mean square error is an inappropriate measure to serve as a basis for comparison. This undermines the claims made for the GFESM.

Suggested Citation

  • JS Armstrong & Robert Fildes, 2004. "Correspondence On the Selection of Error Measures for Comparisons Among Forecasting Methods," General Economics and Teaching 0412002, University Library of Munich, Germany.
    • Handle: RePEc:wpa:wuwpgt:0412002
    Note: Type of Document - pdf; pages: 4
    as

    Download full text from publisher

    File URL: https://econwpa.ub.uni-muenchen.de/econ-wp/get/papers/0412/0412002.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Makridakis, Spyros & Hibon, Michele & Lusk, Ed & Belhadjali, Moncef, 1987. "Confidence intervals: An empirical investigation of the series in the M-competition," International Journal of Forecasting, Elsevier, vol. 3(3-4), pages 489-508.
    2. Zellner, Arnold, 1986. "A tale of forecasting 1001 series : The Bayesian knight strikes again," International Journal of Forecasting, Elsevier, vol. 2(4), pages 491-494.
    3. Clements, M.P. & Hendry, D., 1992. "On the Limitations of Comparing Mean Square Forecast Errors," Economics Series Working Papers 99138, University of Oxford, Department of Economics.
    4. Robert Carbone & JS Armstrong, 2004. "Evaluation of Extrapolative Forecasting Methods: Results of a Survey of Academicians and Practitioners," General Economics and Teaching 0412008, University Library of Munich, Germany.
    5. Murphy, Allan H. & Winkler, Robert L., 1992. "Diagnostic verification of probability forecasts," International Journal of Forecasting, Elsevier, vol. 7(4), pages 435-455, March.
    6. Armstrong, J. Scott & Collopy, Fred, 1992. "Error measures for generalizing about forecasting methods: Empirical comparisons," International Journal of Forecasting, Elsevier, vol. 8(1), pages 69-80, June.
    7. Thompson, Patrick A., 1990. "An MSE statistic for comparing forecast accuracy across series," International Journal of Forecasting, Elsevier, vol. 6(2), pages 219-227, July.
    8. Price, D. H. R. & Sharp, J. A., 1986. "A comparison of the performance of different univariate forecasting methods in a model of capacity acquisition in UK electricity supply," International Journal of Forecasting, Elsevier, vol. 2(3), pages 333-348.
    9. Fildes, Robert, 1992. "The evaluation of extrapolative forecasting methods," International Journal of Forecasting, Elsevier, vol. 8(1), pages 81-98, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wagatha, Matthias, 2007. "Integration, Kointegration und die Langzeitprognose von Kreditausfallzyklen [Integration, Cointegration and Long-Horizont Forecasting of Credit-Default-Cycles]," MPRA Paper 8602, University Library of Munich, Germany.
    2. De Gooijer, Jan G. & Hyndman, Rob J., 2006. "25 years of time series forecasting," International Journal of Forecasting, Elsevier, vol. 22(3), pages 443-473.
    3. Hyndman, Rob J. & Koehler, Anne B., 2006. "Another look at measures of forecast accuracy," International Journal of Forecasting, Elsevier, vol. 22(4), pages 679-688.
    4. Jan G. De Gooijer & Rob J. Hyndman, 2005. "25 Years of IIF Time Series Forecasting: A Selective Review," Monash Econometrics and Business Statistics Working Papers 12/05, Monash University, Department of Econometrics and Business Statistics.
    5. repec:lan:wpaper:470 is not listed on IDEAS
    6. Shah, Chandra, 1997. "Model selection in univariate time series forecasting using discriminant analysis," International Journal of Forecasting, Elsevier, vol. 13(4), pages 489-500, December.
    7. Thury, Gerhard & Witt, Stephen F., 1998. "Forecasting industrial production using structural time series models," Omega, Elsevier, vol. 26(6), pages 751-767, December.
    8. Davydenko, Andrey & Fildes, Robert, 2013. "Measuring forecasting accuracy: The case of judgmental adjustments to SKU-level demand forecasts," International Journal of Forecasting, Elsevier, vol. 29(3), pages 510-522.
    9. Everette S. Gardner, 1999. "Note: Rule-Based Forecasting vs. Damped-Trend Exponential Smoothing," Management Science, INFORMS, vol. 45(8), pages 1169-1176, August.
    10. repec:lan:wpaper:425 is not listed on IDEAS
    11. repec:lan:wpaper:539557 is not listed on IDEAS
    12. repec:lan:wpaper:413 is not listed on IDEAS
    13. Madden, Gary & Tan, Joachim, 2007. "Forecasting telecommunications data with linear models," Telecommunications Policy, Elsevier, vol. 31(1), pages 31-44, February.
    14. Garcia-Ferrer, Antonio & Bujosa-Brun, Marcos, 2000. "Forecasting OECD industrial turning points using unobserved components models with business survey data," International Journal of Forecasting, Elsevier, vol. 16(2), pages 207-227.
    15. Kumar, V. & Leone, Robert P. & Gaskins, John N., 1995. "Aggregate and disaggregate sector forecasting using consumer confidence measures," International Journal of Forecasting, Elsevier, vol. 11(3), pages 361-377, September.
    16. Konrad Bogner & Katharina Liechti & Luzi Bernhard & Samuel Monhart & Massimiliano Zappa, 2018. "Skill of Hydrological Extended Range Forecasts for Water Resources Management in Switzerland," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 32(3), pages 969-984, February.
    17. Garcia-Ferrer, Antonio & Queralt, Ricardo A., 1997. "A note on forecasting international tourism demand in Spain," International Journal of Forecasting, Elsevier, vol. 13(4), pages 539-549, December.
    18. Tashman, Leonard J. & Kruk, Joshua M., 1996. "The use of protocols to select exponential smoothing procedures: A reconsideration of forecasting competitions," International Journal of Forecasting, Elsevier, vol. 12(2), pages 235-253, June.
    19. Bruce G. S. Hardie & Peter S. Fader & Robert Zeithammer, 2003. "Forecasting new product trial in a controlled test market environment," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 22(5), pages 391-410.
    20. Bunn, Derek W. & Taylor, James W., 2001. "Setting accuracy targets for short-term judgemental sales forecasting," International Journal of Forecasting, Elsevier, vol. 17(2), pages 159-169.
    21. Oliver Schaer & Nikolaos Kourentzes & Robert Fildes, 2022. "Predictive competitive intelligence with prerelease online search traffic," Production and Operations Management, Production and Operations Management Society, vol. 31(10), pages 3823-3839, October.
    22. R Fildes & K Nikolopoulos & S F Crone & A A Syntetos, 2008. "Forecasting and operational research: a review," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 59(9), pages 1150-1172, September.
    23. Kim, Sungil & Kim, Heeyoung, 2016. "A new metric of absolute percentage error for intermittent demand forecasts," International Journal of Forecasting, Elsevier, vol. 32(3), pages 669-679.
    24. Mirakyan, Atom & Meyer-Renschhausen, Martin & Koch, Andreas, 2017. "Composite forecasting approach, application for next-day electricity price forecasting," Energy Economics, Elsevier, vol. 66(C), pages 228-237.

    More about this item

    Keywords

    Accuracy Forecast evaluation Loss functions;

    JEL classification:

    • A - General Economics and Teaching

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wpa:wuwpgt:0412002. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: EconWPA (email available below). General contact details of provider: https://econwpa.ub.uni-muenchen.de .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.