Advanced Search
MyIDEAS: Login

Advances in forecasting with neural networks? Empirical evidence from the NN3 competition on time series prediction

Contents:

Author Info

  • Crone, Sven F.
  • Hibon, Michèle
  • Nikolopoulos, Konstantinos

Abstract

This paper reports the results of the NN3 competition, which is a replication of the M3 competition with an extension of the competition towards neural network (NN) and computational intelligence (CI) methods, in order to assess what progress has been made in the 10 years since the M3 competition. Two masked subsets of the M3 monthly industry data, containing 111 and 11 empirical time series respectively, were chosen, controlling for multiple data conditions of time series length (short/long), data patterns (seasonal/non-seasonal) and forecasting horizons (short/medium/long). The relative forecasting accuracy was assessed using the metrics from the M3, together with later extensions of scaled measures, and non-parametric statistical tests. The NN3 competition attracted 59 submissions from NN, CI and statistics, making it the largest CI competition on time series data. Its main findings include: (a) only one NN outperformed the damped trend using the sMAPE, but more contenders outperformed the AutomatANN of the M3; (b) ensembles of CI approaches performed very well, better than combinations of statistical methods; (c) a novel, complex statistical method outperformed all statistical and CI benchmarks; and (d) for the most difficult subset of short and seasonal series, a methodology employing echo state neural networks outperformed all others. The NN3 results highlight the ability of NN to handle complex data, including short and seasonal time series, beyond prior expectations, and thus identify multiple avenues for future research.

Download Info

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
File URL: http://www.sciencedirect.com/science/article/pii/S0169207011000616
Download Restriction: Full text for ScienceDirect subscribers only

As the access to this document is restricted, you may want to look for a different version under "Related research" (further below) or search for a different version of it.

Bibliographic Info

Article provided by Elsevier in its journal International Journal of Forecasting.

Volume (Year): 27 (2011)
Issue (Month): 3 ()
Pages: 635-660

as in new window
Handle: RePEc:eee:intfor:v:27:y:2011:i:3:p:635-660

Contact details of provider:
Web page: http://www.elsevier.com/locate/ijforecast

Related research

Keywords: Time series forecasting; Empirical evaluation; NN3 competition; Artificial neural networks; Computational intelligence;

Other versions of this item:

Find related papers by JEL classification:

References

References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
as in new window
  1. Makridakis, Spyros & Hibon, Michele, 2000. "The M3-Competition: results, conclusions and implications," International Journal of Forecasting, Elsevier, vol. 16(4), pages 451-476.
  2. Athanasopoulos, George & Hyndman, Rob J. & Song, Haiyan & Wu, Doris C., 2011. "The tourism forecasting competition," International Journal of Forecasting, Elsevier, vol. 27(3), pages 822-844.
  3. Chatfield, Chris, 1993. "Neural networks: Forecasting breakthrough or passing fad?," International Journal of Forecasting, Elsevier, vol. 9(1), pages 1-3, April.
  4. Adya, Monica & Armstrong, J. Scott & Collopy, Fred & Kennedy, Miles, 2000. "An application of rule-based forecasting to a situation lacking domain knowledge," International Journal of Forecasting, Elsevier, vol. 16(4), pages 477-484.
  5. Armstrong, J. Scott, 2007. "Statistical significance tests are unnecessary even when properly done and properly interpreted: Reply to commentaries," International Journal of Forecasting, Elsevier, vol. 23(2), pages 335-336.
  6. Qi, Min & Zhang, Guoqiang Peter, 2001. "An investigation of model selection criteria for neural network time series forecasting," European Journal of Operational Research, Elsevier, vol. 132(3), pages 666-680, August.
  7. Timmermann, Allan & Granger, Clive W. J., 2004. "Efficient market hypothesis and forecasting," International Journal of Forecasting, Elsevier, vol. 20(1), pages 15-27.
  8. Syntetos, Aris A. & Nikolopoulos, Konstantinos & Boylan, John E., 2010. "Judging the judges through accuracy-implication metrics: The case of inventory forecasting," International Journal of Forecasting, Elsevier, vol. 26(1), pages 134-143, January.
  9. Makridakis, Spyros & Chatfield, Chris & Hibon, Michele & Lawrence, Michael & Mills, Terence & Ord, Keith & Simmons, LeRoy F., 1993. "The M2-competition: A real-time judgmentally based forecasting study," International Journal of Forecasting, Elsevier, vol. 9(1), pages 5-22, April.
  10. Gene K. Groff, 1973. "Empirical Comparison of Models for Short Range Forecasting," Management Science, INFORMS, vol. 20(1), pages 22-31, September.
  11. Goodwin, Paul, 2007. "Should we be using significance tests in forecasting research?," International Journal of Forecasting, Elsevier, vol. 23(2), pages 333-334.
  12. Fildes, Robert, 2006. "The forecasting journals and their contribution to forecasting research: Citation analysis and expert opinion," International Journal of Forecasting, Elsevier, vol. 22(3), pages 415-432.
  13. Terasvirta, Timo & van Dijk, Dick & Medeiros, Marcelo C., 2005. "Linear models, smooth transition autoregressions, and neural networks for forecasting macroeconomic time series: A re-examination," International Journal of Forecasting, Elsevier, vol. 21(4), pages 755-774.
  14. Billah, Baki & King, Maxwell L. & Snyder, Ralph D. & Koehler, Anne B., 2006. "Exponential smoothing model selection for forecasting," International Journal of Forecasting, Elsevier, vol. 22(2), pages 239-247.
  15. Tim Hill & Marcus O'Connor & William Remus, 1996. "Neural Network Models for Time Series Forecasts," Management Science, INFORMS, vol. 42(7), pages 1082-1092, July.
  16. Zhang, G. Peter & Qi, Min, 2005. "Neural network forecasting for seasonal and trend time series," European Journal of Operational Research, Elsevier, vol. 160(2), pages 501-514, January.
  17. Tashman, Leonard J., 2000. "Out-of-sample tests of forecasting accuracy: an analysis and review," International Journal of Forecasting, Elsevier, vol. 16(4), pages 437-450.
  18. Ord, Keith & Hibon, Michele & Makridakis, Spyros, 2000. "The M3-Competition1," International Journal of Forecasting, Elsevier, vol. 16(4), pages 433-436.
  19. de Menezes, Lilian M. & Nikolaev, Nikolay Y., 2006. "Forecasting with genetically programmed polynomial neural networks," International Journal of Forecasting, Elsevier, vol. 22(2), pages 249-265.
  20. Koning, Alex J. & Franses, Philip Hans & Hibon, Michele & Stekler, H.O., 2005. "The M3 competition: Statistical tests of the results," International Journal of Forecasting, Elsevier, vol. 21(3), pages 397-409.
  21. Hyndman, Rob J. & Koehler, Anne B., 2006. "Another look at measures of forecast accuracy," International Journal of Forecasting, Elsevier, vol. 22(4), pages 679-688.
  22. Curry, Bruce, 2007. "Neural networks and seasonality: Some technical considerations," European Journal of Operational Research, Elsevier, vol. 179(1), pages 267-274, May.
  23. PREMINGER, Arie & FRANCK, Raphael, 2005. "Forecasting exchange rates: a robust regression approach," CORE Discussion Papers 2005025, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
  24. Goodrich, Robert L., 2000. "The Forecast Pro methodology," International Journal of Forecasting, Elsevier, vol. 16(4), pages 533-535.
  25. Fildes, Robert, 1992. "The evaluation of extrapolative forecasting methods," International Journal of Forecasting, Elsevier, vol. 8(1), pages 81-98, June.
  26. Armstrong, J. Scott, 2006. "Findings from evidence-based forecasting: Methods for reducing forecast error," International Journal of Forecasting, Elsevier, vol. 22(3), pages 583-598.
  27. Assimakopoulos, V. & Nikolopoulos, K., 2000. "The theta model: a decomposition approach to forecasting," International Journal of Forecasting, Elsevier, vol. 16(4), pages 521-530.
  28. Armstrong, J. Scott, 2007. "Significance tests harm progress in forecasting," International Journal of Forecasting, Elsevier, vol. 23(2), pages 321-327.
  29. Balkin, Sandy D. & Ord, J. Keith, 2000. "Automatic neural network modeling for univariate time series," International Journal of Forecasting, Elsevier, vol. 16(4), pages 509-515.
  30. Fildes, Robert & Hibon, Michele & Makridakis, Spyros & Meade, Nigel, 1998. "Generalising about univariate forecasting methods: further empirical evidence," International Journal of Forecasting, Elsevier, vol. 14(3), pages 339-358, September.
  31. Goodwin, Paul & Lawton, Richard, 1999. "On the asymmetry of the symmetric MAPE," International Journal of Forecasting, Elsevier, vol. 15(4), pages 405-408, October.
Full references (including those not matched with items on IDEAS)

Citations

Lists

This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

Statistics

Access and download statistics

Corrections

When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:27:y:2011:i:3:p:635-660. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Zhang, Lei).

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.