IDEAS home Printed from https://ideas.repec.org/p/msh/ebswps/2019-21.html
   My bibliography  Save this paper

Feature-based Forecast-Model Performance Prediction

Author

Listed:
  • Thiyanga S. Talagala

    ()

  • Feng Li

    ()

  • Yanfei Kang

    ()

Abstract

This paper introduces a novel meta-learning algorithm for time series forecasting. The efficient Bayesian multivariate surface regression approach is used to model forecast error as a function of features calculated from the time series. The minimum predicted forecast error is then used to identify an individual model or combination of models to produce forecasts. In general, the performance of any meta-learner strongly depends on the reference dataset used to train the model. We further examine the feasibility of using GRATIS (a feature-based time series simulation approach) in generating a realistic time series collection to obtain a diverse collection of time series for our reference set. The proposed framework is tested using the M4 competition data and is compared against several benchmarks and other commonly used forecasting approaches. The new approach obtains performance comparable to the second and the third rankings of the M4 competition.

Suggested Citation

  • Thiyanga S. Talagala & Feng Li & Yanfei Kang, 2019. "Feature-based Forecast-Model Performance Prediction," Monash Econometrics and Business Statistics Working Papers 21/19, Monash University, Department of Econometrics and Business Statistics.
  • Handle: RePEc:msh:ebswps:2019-21
    as

    Download full text from publisher

    File URL: https://www.monash.edu/business/ebs/research/publications/ebs/wp21-2019.pdf
    Download Restriction: no

    References listed on IDEAS

    as
    1. Shah, Chandra, 1997. "Model selection in univariate time series forecasting using discriminant analysis," International Journal of Forecasting, Elsevier, vol. 13(4), pages 489-500, December.
    2. Kang, Yanfei & Hyndman, Rob J. & Smith-Miles, Kate, 2017. "Visualising forecasting algorithm performance using time series instance spaces," International Journal of Forecasting, Elsevier, vol. 33(2), pages 345-358.
    3. Fred Collopy & J. Scott Armstrong, 1992. "Rule-Based Forecasting: Development and Validation of an Expert Systems Approach to Combining Time Series Extrapolations," Management Science, INFORMS, vol. 38(10), pages 1394-1414, October.
    4. Montero-Manso, Pablo & Athanasopoulos, George & Hyndman, Rob J. & Talagala, Thiyanga S., 2020. "FFORMA: Feature-based forecast model averaging," International Journal of Forecasting, Elsevier, vol. 36(1), pages 86-92.
    5. Tashman, Leonard J. & Leach, Michael L., 1991. "Automatic forecasting software: A survey and evaluation," International Journal of Forecasting, Elsevier, vol. 7(2), pages 209-230, August.
    6. Adya, Monica & Collopy, Fred & Armstrong, J. Scott & Kennedy, Miles, 2001. "Automatic identification of time series features for rule-based forecasting," International Journal of Forecasting, Elsevier, vol. 17(2), pages 143-157.
    7. Makridakis, Spyros & Spiliotis, Evangelos & Assimakopoulos, Vassilios, 2018. "The M4 Competition: Results, findings, conclusion and way forward," International Journal of Forecasting, Elsevier, vol. 34(4), pages 802-808.
    8. Thiyanga S Talagala & Rob J Hyndman & George Athanasopoulos, 2018. "Meta-learning how to forecast time series," Monash Econometrics and Business Statistics Working Papers 6/18, Monash University, Department of Econometrics and Business Statistics.
    9. Petropoulos, Fotios & Makridakis, Spyros & Assimakopoulos, Vassilios & Nikolopoulos, Konstantinos, 2014. "‘Horses for Courses’ in demand forecasting," European Journal of Operational Research, Elsevier, vol. 237(1), pages 152-163.
    10. Feng Li & Mattias Villani, 2013. "Efficient Bayesian Multivariate Surface Regression," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 40(4), pages 706-723, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Thiyanga S Talagala & Rob J Hyndman & George Athanasopoulos, 2018. "Meta-learning how to forecast time series," Monash Econometrics and Business Statistics Working Papers 6/18, Monash University, Department of Econometrics and Business Statistics.
    2. Makridakis, Spyros & Hyndman, Rob J. & Petropoulos, Fotios, 2020. "Forecasting in social settings: The state of the art," International Journal of Forecasting, Elsevier, vol. 36(1), pages 15-28.
    3. Fildes, Robert & Petropoulos, Fotios, 2015. "Simple versus complex selection rules for forecasting many time series," Journal of Business Research, Elsevier, vol. 68(8), pages 1692-1701.
    4. Fildes, Robert & Petropoulos, Fotios, 2013. "An evaluation of simple forecasting model selection rules," MPRA Paper 51772, University Library of Munich, Germany.
    5. Spiliotis, Evangelos & Nikolopoulos, Konstantinos & Assimakopoulos, Vassilios, 2019. "Tales from tails: On the empirical distributions of forecasting errors and their implication to risk," International Journal of Forecasting, Elsevier, vol. 35(2), pages 687-698.
    6. Robert Fildes & Gary Madden & Joachim Tan, 2007. "Optimal forecasting model selection and data characteristics," Applied Financial Economics, Taylor & Francis Journals, vol. 17(15), pages 1251-1264.
    7. Yanfei Kang & Rob J Hyndman & Feng Li, 2018. "Efficient generation of time series with diverse and controllable characteristics," Monash Econometrics and Business Statistics Working Papers 15/18, Monash University, Department of Econometrics and Business Statistics.
    8. Spiliotis, Evangelos & Kouloumos, Andreas & Assimakopoulos, Vassilios & Makridakis, Spyros, 2020. "Are forecasting competitions data representative of the reality?," International Journal of Forecasting, Elsevier, vol. 36(1), pages 37-53.
    9. Petropoulos, Fotios & Makridakis, Spyros & Assimakopoulos, Vassilios & Nikolopoulos, Konstantinos, 2014. "‘Horses for Courses’ in demand forecasting," European Journal of Operational Research, Elsevier, vol. 237(1), pages 152-163.
    10. Gardner, Everette Jr., 2006. "Exponential smoothing: The state of the art--Part II," International Journal of Forecasting, Elsevier, vol. 22(4), pages 637-666.
    11. Smyl, Slawek, 2020. "A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting," International Journal of Forecasting, Elsevier, vol. 36(1), pages 75-85.
    12. Hewamalage, Hansika & Bergmeir, Christoph & Bandara, Kasun, 2021. "Recurrent Neural Networks for Time Series Forecasting: Current status and future directions," International Journal of Forecasting, Elsevier, vol. 37(1), pages 388-427.
    13. Huber, Jakob & Stuckenschmidt, Heiner, 2020. "Daily retail demand forecasting using machine learning with emphasis on calendric special days," International Journal of Forecasting, Elsevier, vol. 36(4), pages 1420-1438.
    14. Kusters, Ulrich & McCullough, B.D. & Bell, Michael, 2006. "Forecasting software: Past, present and future," International Journal of Forecasting, Elsevier, vol. 22(3), pages 599-615.
    15. Han, Weiwei & Wang, Xun & Petropoulos, Fotios & Wang, Jing, 2019. "Brain imaging and forecasting: Insights from judgmental model selection," Omega, Elsevier, vol. 87(C), pages 1-9.
    16. Shaub, David, 2020. "Fast and accurate yearly time series forecasting with forecast combinations," International Journal of Forecasting, Elsevier, vol. 36(1), pages 116-120.
    17. Arvan, Meysam & Fahimnia, Behnam & Reisi, Mohsen & Siemsen, Enno, 2019. "Integrating human judgement into quantitative forecasting methods: A review," Omega, Elsevier, vol. 86(C), pages 237-252.
    18. Makridakis, Spyros & Spiliotis, Evangelos & Assimakopoulos, Vassilios, 2020. "The M4 Competition: 100,000 time series and 61 forecasting methods," International Journal of Forecasting, Elsevier, vol. 36(1), pages 54-74.
    19. JS Armstrong, 2004. "Forecasting for Environmental Decision Making," General Economics and Teaching 0412023, University Library of Munich, Germany.
    20. Makridakis, Spyros & Spiliotis, Evangelos & Assimakopoulos, Vassilios, 2018. "The M4 Competition: Results, findings, conclusion and way forward," International Journal of Forecasting, Elsevier, vol. 34(4), pages 802-808.

    More about this item

    Keywords

    tme series; meta-learning; mixture autoregressive models; surface regression; M4 competition;
    All these keywords.

    JEL classification:

    • C10 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - General
    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General
    • C22 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:msh:ebswps:2019-21. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Dr Xibin Zhang). General contact details of provider: http://edirc.repec.org/data/dxmonau.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.