IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v27y2011i3p635-660.html
   My bibliography  Save this article

Advances in forecasting with neural networks? Empirical evidence from the NN3 competition on time series prediction

Author

Listed:
  • Crone, Sven F.
  • Hibon, Michèle
  • Nikolopoulos, Konstantinos

Abstract

This paper reports the results of the NN3 competition, which is a replication of the M3 competition with an extension of the competition towards neural network (NN) and computational intelligence (CI) methods, in order to assess what progress has been made in the 10 years since the M3 competition. Two masked subsets of the M3 monthly industry data, containing 111 and 11 empirical time series respectively, were chosen, controlling for multiple data conditions of time series length (short/long), data patterns (seasonal/non-seasonal) and forecasting horizons (short/medium/long). The relative forecasting accuracy was assessed using the metrics from the M3, together with later extensions of scaled measures, and non-parametric statistical tests. The NN3 competition attracted 59 submissions from NN, CI and statistics, making it the largest CI competition on time series data. Its main findings include: (a) only one NN outperformed the damped trend using the sMAPE, but more contenders outperformed the AutomatANN of the M3; (b) ensembles of CI approaches performed very well, better than combinations of statistical methods; (c) a novel, complex statistical method outperformed all statistical and CI benchmarks; and (d) for the most difficult subset of short and seasonal series, a methodology employing echo state neural networks outperformed all others. The NN3 results highlight the ability of NN to handle complex data, including short and seasonal time series, beyond prior expectations, and thus identify multiple avenues for future research.

Suggested Citation

  • Crone, Sven F. & Hibon, Michèle & Nikolopoulos, Konstantinos, 2011. "Advances in forecasting with neural networks? Empirical evidence from the NN3 competition on time series prediction," International Journal of Forecasting, Elsevier, vol. 27(3), pages 635-660.
  • Handle: RePEc:eee:intfor:v:27:y:2011:i:3:p:635-660
    DOI: 10.1016/j.ijforecast.2011.04.001
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0169207011000616
    Download Restriction: Full text for ScienceDirect subscribers only

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Armstrong, J. Scott, 2007. "Significance Tests Harm Progress in Forecasting," MPRA Paper 81664, University Library of Munich, Germany.
    2. Makridakis, Spyros & Hibon, Michele, 2000. "The M3-Competition: results, conclusions and implications," International Journal of Forecasting, Elsevier, vol. 16(4), pages 451-476.
    3. Terasvirta, Timo & van Dijk, Dick & Medeiros, Marcelo C., 2005. "Linear models, smooth transition autoregressions, and neural networks for forecasting macroeconomic time series: A re-examination," International Journal of Forecasting, Elsevier, vol. 21(4), pages 755-774.
    4. Balkin, Sandy D. & Ord, J. Keith, 2000. "Automatic neural network modeling for univariate time series," International Journal of Forecasting, Elsevier, vol. 16(4), pages 509-515.
    5. Fildes, Robert & Hibon, Michele & Makridakis, Spyros & Meade, Nigel, 1998. "Generalising about univariate forecasting methods: further empirical evidence," International Journal of Forecasting, Elsevier, vol. 14(3), pages 339-358, September.
    6. de Menezes, Lilian M. & Nikolaev, Nikolay Y., 2006. "Forecasting with genetically programmed polynomial neural networks," International Journal of Forecasting, Elsevier, vol. 22(2), pages 249-265.
    7. Timmermann, Allan & Granger, Clive W. J., 2004. "Efficient market hypothesis and forecasting," International Journal of Forecasting, Elsevier, vol. 20(1), pages 15-27.
    8. Goodwin, Paul, 2007. "Should we be using significance tests in forecasting research?," International Journal of Forecasting, Elsevier, vol. 23(2), pages 333-334.
    9. Tashman, Leonard J., 2000. "Out-of-sample tests of forecasting accuracy: an analysis and review," International Journal of Forecasting, Elsevier, vol. 16(4), pages 437-450.
    10. Preminger, Arie & Franck, Raphael, 2007. "Forecasting exchange rates: A robust regression approach," International Journal of Forecasting, Elsevier, vol. 23(1), pages 71-84.
    11. Gene K. Groff, 1973. "Empirical Comparison of Models for Short Range Forecasting," Management Science, INFORMS, vol. 20(1), pages 22-31, September.
    12. Assimakopoulos, V. & Nikolopoulos, K., 2000. "The theta model: a decomposition approach to forecasting," International Journal of Forecasting, Elsevier, vol. 16(4), pages 521-530.
    13. Goodrich, Robert L., 2000. "The Forecast Pro methodology," International Journal of Forecasting, Elsevier, vol. 16(4), pages 533-535.
    14. Billah, Baki & King, Maxwell L. & Snyder, Ralph D. & Koehler, Anne B., 2006. "Exponential smoothing model selection for forecasting," International Journal of Forecasting, Elsevier, vol. 22(2), pages 239-247.
    15. Armstrong, J. Scott, 2007. "Significance tests harm progress in forecasting," International Journal of Forecasting, Elsevier, vol. 23(2), pages 321-327.
    16. Athanasopoulos, George & Hyndman, Rob J. & Song, Haiyan & Wu, Doris C., 2011. "The tourism forecasting competition," International Journal of Forecasting, Elsevier, vol. 27(3), pages 822-844, July.
    17. Curry, Bruce, 2007. "Neural networks and seasonality: Some technical considerations," European Journal of Operational Research, Elsevier, vol. 179(1), pages 267-274, May.
    18. Qi, Min & Zhang, Guoqiang Peter, 2001. "An investigation of model selection criteria for neural network time series forecasting," European Journal of Operational Research, Elsevier, vol. 132(3), pages 666-680, August.
    19. Tim Hill & Marcus O'Connor & William Remus, 1996. "Neural Network Models for Time Series Forecasts," Management Science, INFORMS, vol. 42(7), pages 1082-1092, July.
    20. Makridakis, Spyros & Chatfield, Chris & Hibon, Michele & Lawrence, Michael & Mills, Terence & Ord, Keith & Simmons, LeRoy F., 1993. "The M2-competition: A real-time judgmentally based forecasting study," International Journal of Forecasting, Elsevier, vol. 9(1), pages 5-22, April.
    21. Hyndman, Rob J. & Koehler, Anne B., 2006. "Another look at measures of forecast accuracy," International Journal of Forecasting, Elsevier, vol. 22(4), pages 679-688.
    22. Zhang, G. Peter & Qi, Min, 2005. "Neural network forecasting for seasonal and trend time series," European Journal of Operational Research, Elsevier, vol. 160(2), pages 501-514, January.
    23. Chatfield, Chris, 1993. "Neural networks: Forecasting breakthrough or passing fad?," International Journal of Forecasting, Elsevier, vol. 9(1), pages 1-3, April.
    24. Syntetos, Aris A. & Nikolopoulos, Konstantinos & Boylan, John E., 2010. "Judging the judges through accuracy-implication metrics: The case of inventory forecasting," International Journal of Forecasting, Elsevier, vol. 26(1), pages 134-143, January.
    25. Armstrong, J. Scott, 2007. "Statistical significance tests are unnecessary even when properly done and properly interpreted: Reply to commentaries," International Journal of Forecasting, Elsevier, vol. 23(2), pages 335-336.
    26. Fildes, Robert, 2006. "The forecasting journals and their contribution to forecasting research: Citation analysis and expert opinion," International Journal of Forecasting, Elsevier, vol. 22(3), pages 415-432.
    27. Koning, Alex J. & Franses, Philip Hans & Hibon, Michele & Stekler, H.O., 2005. "The M3 competition: Statistical tests of the results," International Journal of Forecasting, Elsevier, vol. 21(3), pages 397-409.
    28. Goodwin, Paul & Lawton, Richard, 1999. "On the asymmetry of the symmetric MAPE," International Journal of Forecasting, Elsevier, vol. 15(4), pages 405-408, October.
    29. Adya, Monica & Armstrong, J. Scott & Collopy, Fred & Kennedy, Miles, 2000. "An application of rule-based forecasting to a situation lacking domain knowledge," International Journal of Forecasting, Elsevier, vol. 16(4), pages 477-484.
    30. Fildes, Robert, 1992. "The evaluation of extrapolative forecasting methods," International Journal of Forecasting, Elsevier, vol. 8(1), pages 81-98, June.
    31. Ord, Keith & Hibon, Michele & Makridakis, Spyros, 2000. "The M3-Competition1," International Journal of Forecasting, Elsevier, vol. 16(4), pages 433-436.
    32. Armstrong, J. Scott, 2006. "Findings from evidence-based forecasting: Methods for reducing forecast error," International Journal of Forecasting, Elsevier, vol. 22(3), pages 583-598.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. repec:eee:ejores:v:264:y:2018:i:3:p:967-977 is not listed on IDEAS
    2. Petropoulos, Fotios & Makridakis, Spyros & Assimakopoulos, Vassilios & Nikolopoulos, Konstantinos, 2014. "‘Horses for Courses’ in demand forecasting," European Journal of Operational Research, Elsevier, vol. 237(1), pages 152-163.
    3. Green, Kesten C. & Armstrong, J. Scott, 2015. "Simple versus complex forecasting: The evidence," Journal of Business Research, Elsevier, vol. 68(8), pages 1678-1685.
    4. Advait Sarkar & Neal Lathia & Cecilia Mascolo, 2015. "Comparing cities’ cycling patterns using online shared bicycle maps," Transportation, Springer, vol. 42(4), pages 541-559, July.
    5. repec:eee:intfor:v:33:y:2017:i:4:p:864-877 is not listed on IDEAS
    6. Oscar Claveria & Enric Monte & Salvador Torra, 2016. "Modelling cross-dependencies between Spain’s regional tourism markets with an extension of the Gaussian process regression model," SERIEs: Journal of the Spanish Economic Association, Springer;Spanish Economic Association, vol. 7(3), pages 341-357, August.
    7. Barrow, Devon K. & Crone, Sven F., 2016. "A comparison of AdaBoost algorithms for time series forecast combination," International Journal of Forecasting, Elsevier, vol. 32(4), pages 1103-1119.
    8. repec:eee:eneeco:v:66:y:2017:i:c:p:228-237 is not listed on IDEAS
    9. Ruhnau, Oliver & Hennig, Patrick & Madlener, Reinhard, 2015. "Economic Implications of Enhanced Forecast Accuracy: The Case of Photovoltaic Feed-In Forecasts," FCN Working Papers 6/2015, E.ON Energy Research Center, Future Energy Consumer Needs and Behavior (FCN).
    10. Barrow, Devon K. & Crone, Sven F., 2016. "Cross-validation aggregation for combining autoregressive neural network forecasts," International Journal of Forecasting, Elsevier, vol. 32(4), pages 1120-1137.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:27:y:2011:i:3:p:635-660. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Dana Niculescu). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.