IDEAS home Printed from https://ideas.repec.org/
MyIDEAS: Login to save this article or follow this journal

Advances in forecasting with neural networks? Empirical evidence from the NN3 competition on time series prediction

  • Crone, Sven F.
  • Hibon, Michèle
  • Nikolopoulos, Konstantinos

This paper reports the results of the NN3 competition, which is a replication of the M3 competition with an extension of the competition towards neural network (NN) and computational intelligence (CI) methods, in order to assess what progress has been made in the 10 years since the M3 competition. Two masked subsets of the M3 monthly industry data, containing 111 and 11 empirical time series respectively, were chosen, controlling for multiple data conditions of time series length (short/long), data patterns (seasonal/non-seasonal) and forecasting horizons (short/medium/long). The relative forecasting accuracy was assessed using the metrics from the M3, together with later extensions of scaled measures, and non-parametric statistical tests. The NN3 competition attracted 59 submissions from NN, CI and statistics, making it the largest CI competition on time series data. Its main findings include: (a) only one NN outperformed the damped trend using the sMAPE, but more contenders outperformed the AutomatANN of the M3; (b) ensembles of CI approaches performed very well, better than combinations of statistical methods; (c) a novel, complex statistical method outperformed all statistical and CI benchmarks; and (d) for the most difficult subset of short and seasonal series, a methodology employing echo state neural networks outperformed all others. The NN3 results highlight the ability of NN to handle complex data, including short and seasonal time series, beyond prior expectations, and thus identify multiple avenues for future research.

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.

File URL: http://www.sciencedirect.com/science/article/pii/S0169207011000616
Download Restriction: Full text for ScienceDirect subscribers only

As the access to this document is restricted, you may want to look for a different version under "Related research" (further below) or search for a different version of it.

Article provided by Elsevier in its journal International Journal of Forecasting.

Volume (Year): 27 (2011)
Issue (Month): 3 (July)
Pages: 635-660

as
in new window

Handle: RePEc:eee:intfor:v:27:y::i:3:p:635-660
Contact details of provider: Web page: http://www.elsevier.com/locate/ijforecast

References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:

as in new window
  1. Qi, Min & Zhang, Guoqiang Peter, 2001. "An investigation of model selection criteria for neural network time series forecasting," European Journal of Operational Research, Elsevier, vol. 132(3), pages 666-680, August.
  2. Ord, Keith & Hibon, Michele & Makridakis, Spyros, 2000. "The M3-Competition1," International Journal of Forecasting, Elsevier, vol. 16(4), pages 433-436.
  3. Billah, Baki & King, Maxwell L. & Snyder, Ralph D. & Koehler, Anne B., 2006. "Exponential smoothing model selection for forecasting," International Journal of Forecasting, Elsevier, vol. 22(2), pages 239-247.
  4. Goodrich, Robert L., 2000. "The Forecast Pro methodology," International Journal of Forecasting, Elsevier, vol. 16(4), pages 533-535.
  5. Assimakopoulos, V. & Nikolopoulos, K., 2000. "The theta model: a decomposition approach to forecasting," International Journal of Forecasting, Elsevier, vol. 16(4), pages 521-530.
  6. Goodwin, Paul, 2007. "Should we be using significance tests in forecasting research?," International Journal of Forecasting, Elsevier, vol. 23(2), pages 333-334.
  7. Curry, Bruce, 2007. "Neural networks and seasonality: Some technical considerations," European Journal of Operational Research, Elsevier, vol. 179(1), pages 267-274, May.
  8. Makridakis, Spyros & Hibon, Michele, 2000. "The M3-Competition: results, conclusions and implications," International Journal of Forecasting, Elsevier, vol. 16(4), pages 451-476.
  9. Hyndman, Rob J. & Koehler, Anne B., 2006. "Another look at measures of forecast accuracy," International Journal of Forecasting, Elsevier, vol. 22(4), pages 679-688.
  10. Fildes, Robert, 2006. "The forecasting journals and their contribution to forecasting research: Citation analysis and expert opinion," International Journal of Forecasting, Elsevier, vol. 22(3), pages 415-432.
  11. Fildes, Robert & Hibon, Michele & Makridakis, Spyros & Meade, Nigel, 1998. "Generalising about univariate forecasting methods: further empirical evidence," International Journal of Forecasting, Elsevier, vol. 14(3), pages 339-358, September.
  12. Armstrong, J. Scott, 2006. "Findings from evidence-based forecasting: Methods for reducing forecast error," International Journal of Forecasting, Elsevier, vol. 22(3), pages 583-598.
  13. Timmermann, Allan & Granger, Clive W. J., 2004. "Efficient market hypothesis and forecasting," International Journal of Forecasting, Elsevier, vol. 20(1), pages 15-27.
  14. Adya, Monica & Armstrong, J. Scott & Collopy, Fred & Kennedy, Miles, 2000. "An application of rule-based forecasting to a situation lacking domain knowledge," International Journal of Forecasting, Elsevier, vol. 16(4), pages 477-484.
  15. Syntetos, Aris A. & Nikolopoulos, Konstantinos & Boylan, John E., 2010. "Judging the judges through accuracy-implication metrics: The case of inventory forecasting," International Journal of Forecasting, Elsevier, vol. 26(1), pages 134-143, January.
  16. de Menezes, Lilian M. & Nikolaev, Nikolay Y., 2006. "Forecasting with genetically programmed polynomial neural networks," International Journal of Forecasting, Elsevier, vol. 22(2), pages 249-265.
  17. Gene K. Groff, 1973. "Empirical Comparison of Models for Short Range Forecasting," Management Science, INFORMS, vol. 20(1), pages 22-31, September.
  18. Fildes, Robert, 1992. "The evaluation of extrapolative forecasting methods," International Journal of Forecasting, Elsevier, vol. 8(1), pages 81-98, June.
  19. Tim Hill & Marcus O'Connor & William Remus, 1996. "Neural Network Models for Time Series Forecasts," Management Science, INFORMS, vol. 42(7), pages 1082-1092, July.
  20. Koning, Alex J. & Franses, Philip Hans & Hibon, Michele & Stekler, H.O., 2005. "The M3 competition: Statistical tests of the results," International Journal of Forecasting, Elsevier, vol. 21(3), pages 397-409.
  21. Terasvirta, Timo & van Dijk, Dick & Medeiros, Marcelo C., 2005. "Linear models, smooth transition autoregressions, and neural networks for forecasting macroeconomic time series: A re-examination," International Journal of Forecasting, Elsevier, vol. 21(4), pages 755-774.
  22. Armstrong, J. Scott, 2007. "Significance tests harm progress in forecasting," International Journal of Forecasting, Elsevier, vol. 23(2), pages 321-327.
  23. George Athanasopoulos & Rob J Hyndman & Haiyan Song & Doris C Wu, 2008. "The tourism forecasting competition," Monash Econometrics and Business Statistics Working Papers 10/08, Monash University, Department of Econometrics and Business Statistics, revised Oct 2009.
  24. PREMINGER, Arie & FRANCK, Raphael, . "Forecasting exchange rates: a robust regression approach," CORE Discussion Papers RP 1917, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
  25. Makridakis, Spyros & Chatfield, Chris & Hibon, Michele & Lawrence, Michael & Mills, Terence & Ord, Keith & Simmons, LeRoy F., 1993. "The M2-competition: A real-time judgmentally based forecasting study," International Journal of Forecasting, Elsevier, vol. 9(1), pages 5-22, April.
  26. Balkin, Sandy D. & Ord, J. Keith, 2000. "Automatic neural network modeling for univariate time series," International Journal of Forecasting, Elsevier, vol. 16(4), pages 509-515.
  27. Tashman, Leonard J., 2000. "Out-of-sample tests of forecasting accuracy: an analysis and review," International Journal of Forecasting, Elsevier, vol. 16(4), pages 437-450.
  28. Goodwin, Paul & Lawton, Richard, 1999. "On the asymmetry of the symmetric MAPE," International Journal of Forecasting, Elsevier, vol. 15(4), pages 405-408, October.
  29. Armstrong, J. Scott, 2007. "Statistical significance tests are unnecessary even when properly done and properly interpreted: Reply to commentaries," International Journal of Forecasting, Elsevier, vol. 23(2), pages 335-336.
  30. Chatfield, Chris, 1993. "Neural networks: Forecasting breakthrough or passing fad?," International Journal of Forecasting, Elsevier, vol. 9(1), pages 1-3, April.
  31. Zhang, G. Peter & Qi, Min, 2005. "Neural network forecasting for seasonal and trend time series," European Journal of Operational Research, Elsevier, vol. 160(2), pages 501-514, January.
Full references (including those not matched with items on IDEAS)

This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:27:y::i:3:p:635-660. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Zhang, Lei)

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.

This information is provided to you by IDEAS at the Research Division of the Federal Reserve Bank of St. Louis using RePEc data.