IDEAS home Printed from
   My bibliography  Save this paper

Cross-validation based forecasting method: a machine learning approach


  • Pinto, Jeronymo Marcondes
  • Marçal, Emerson Fernandes


Our paper aims to evaluate two novel methods on selecting the best forecasting model or its combination based on a Machine Learning approach. The methods are based on the selection of the ”best” model, or combination of models, by crossvalidation technique, from a set of possible models. The first one is based on the seminal paper of Granger-Bates (1969) but weights are estimated by a process of cross-validation applied on the training set. The second one selects the model with the best forecasting performance in the process described above, which we called CvML (Cross-Validation Machine Learning Method). The following models are used: exponential smoothing, SARIMA, artificial neural networks and Threshold autoregression (TAR). Model specification is chosen by R packages: forecast and TSA. Both methods – CvML and MGB - are applied to these models to generate forecasts from one up to twelve periods ahead. Frequency of data is monthly. We run the forecasts exercise to the following to monthly series of Industrial Product Indices for seven countries: Canada, Brazil, Belgium, Germany, Portugal, UK and USA. The data was collected at OECD data, with 504 observations. We choose Average Forecast Combination, Granger Bates Method, MCS model, Naive and Seasonal Naive Model as benchmarks.Our results suggest that MGB did not performed well. However, CvML had a lower mean absolute error for most of countries and forecast horizons, particularly at longer horizons, surpassing all the proposed benchmarks. Similar results hold for absolute mean forecast error.

Suggested Citation

  • Pinto, Jeronymo Marcondes & Marçal, Emerson Fernandes, 2019. "Cross-validation based forecasting method: a machine learning approach," Textos para discussão 498, FGV EESP - Escola de Economia de São Paulo, Fundação Getulio Vargas (Brazil).
  • Handle: RePEc:fgv:eesptd:498

    Download full text from publisher

    File URL:
    Download Restriction: no

    References listed on IDEAS

    1. Castle, Jennifer L. & Clements, Michael P. & Hendry, David F., 2015. "Robust approaches to forecasting," International Journal of Forecasting, Elsevier, vol. 31(1), pages 99-112.
    2. Hyndman, Rob J. & Khandakar, Yeasmin, 2008. "Automatic Time Series Forecasting: The forecast Package for R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 27(i03).
    3. Barbara Rossi, 2013. "Exchange Rate Predictability," Journal of Economic Literature, American Economic Association, vol. 51(4), pages 1063-1119, December.
    4. Hsiao, Cheng & Wan, Shui Ki, 2014. "Is there an optimal forecast combination?," Journal of Econometrics, Elsevier, vol. 178(P2), pages 294-309.
    5. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    6. Peter R. Hansen & Asger Lunde & James M. Nason, 2011. "The Model Confidence Set," Econometrica, Econometric Society, vol. 79(2), pages 453-497, March.
    7. Garcia, Márcio G.P. & Medeiros, Marcelo C. & Vasconcelos, Gabriel F.R., 2017. "Real-time inflation forecasting with high-dimensional models: The case of Brazil," International Journal of Forecasting, Elsevier, vol. 33(3), pages 679-693.
    8. Chan, Felix & Pauwels, Laurent L., 2018. "Some theoretical results on forecast combinations," International Journal of Forecasting, Elsevier, vol. 34(1), pages 64-74.
    9. Alzahrani, Naif & Neal, Peter & Spencer, Simon E.F. & McKinley, Trevelyan J. & Touloupou, Panayiota, 2018. "Model selection for time series of count data," Computational Statistics & Data Analysis, Elsevier, vol. 122(C), pages 33-44.
    10. Castle, Jennifer L. & Clements, Michael P. & Hendry, David F., 2013. "Forecasting by factors, by variables, by both or neither?," Journal of Econometrics, Elsevier, vol. 177(2), pages 305-319.
    11. Harding, Don & Pagan, Adrian, 2002. "Dissecting the cycle: a methodological investigation," Journal of Monetary Economics, Elsevier, vol. 49(2), pages 365-381, March.
    12. Bergmeir, Christoph & Costantini, Mauro & Benítez, José M., 2014. "On the usefulness of cross-validation for directional forecast evaluation," Computational Statistics & Data Analysis, Elsevier, vol. 76(C), pages 132-143.
    13. Christoph Bergmeir & Rob J Hyndman & Bonsoo Koo, 2015. "A Note on the Validity of Cross-Validation for Evaluating Time Series Prediction," Monash Econometrics and Business Statistics Working Papers 10/15, Monash University, Department of Econometrics and Business Statistics.
    14. Racine, Jeff, 2000. "Consistent cross-validatory model-selection for dependent data: hv-block cross-validation," Journal of Econometrics, Elsevier, vol. 99(1), pages 39-61, November.
    15. Gerhard Bry & Charlotte Boschan, 1971. "Programmed Selection of Cyclical Turning Points," NBER Chapters, in: Cyclical Analysis of Time Series: Selected Procedures and Computer Programs, pages 7-63, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gary S. Anderson & Alena Audzeyeva, 2019. "A Coherent Framework for Predicting Emerging Market Credit Spreads with Support Vector Regression," Finance and Economics Discussion Series 2019-074, Board of Governors of the Federal Reserve System (U.S.).
    2. Afanasyev, Dmitriy O. & Fedorova, Elena A., 2019. "On the impact of outlier filtering on the electricity price forecasting accuracy," Applied Energy, Elsevier, vol. 236(C), pages 196-210.
    3. Mei-Li Shen & Cheng-Feng Lee & Hsiou-Hsiang Liu & Po-Yin Chang & Cheng-Hong Yang, 2021. "An Effective Hybrid Approach for Forecasting Currency Exchange Rates," Sustainability, MDPI, Open Access Journal, vol. 13(5), pages 1-29, March.
    4. Mundt, Philipp & Alfarano, Simone & Milaković, Mishael, 2020. "Exploiting ergodicity in forecasts of corporate profitability," Journal of Economic Dynamics and Control, Elsevier, vol. 111(C).
    5. Pierre Guérin & Danilo Leiva-Leon & Massimiliano Marcellino, 2020. "Markov-Switching Three-Pass Regression Filter," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 38(2), pages 285-302, April.
    6. Ricardo P. Masini & Marcelo C. Medeiros & Eduardo F. Mendes, 2020. "Machine Learning Advances for Time Series Forecasting," Papers 2012.12802,, revised Apr 2021.
    7. Emerson Fernandes Marçal & Eli Hadad Junior, 2016. "Is It Possible to Beat the Random Walk Model in Exchange Rate Forecasting? More Evidence for Brazilian Case," Brazilian Review of Finance, Brazilian Society of Finance, vol. 14(1), pages 65-88.
    8. Ericsson, Neil R., 2017. "How biased are U.S. government forecasts of the federal debt?," International Journal of Forecasting, Elsevier, vol. 33(2), pages 543-559.
    9. Jack Fosten, 2016. "Forecast evaluation with factor-augmented models," University of East Anglia School of Economics Working Paper Series 2016-05, School of Economics, University of East Anglia, Norwich, UK..
    10. Spiliotis, Evangelos & Nikolopoulos, Konstantinos & Assimakopoulos, Vassilios, 2019. "Tales from tails: On the empirical distributions of forecasting errors and their implication to risk," International Journal of Forecasting, Elsevier, vol. 35(2), pages 687-698.
    11. Jack Fosten & Daniel Gutknecht, 2021. "Horizon confidence sets," Empirical Economics, Springer, vol. 61(2), pages 667-692, August.
    12. Drachal, Krzysztof, 2019. "Forecasting prices of selected metals with Bayesian data-rich models," Resources Policy, Elsevier, vol. 64(C).
    13. Zongwu Cai & Chaoqun Ma & Xianhua Mi, 2020. "Realized Volatility Forecasting Based on Dynamic Quantile Model Averaging," WORKING PAPERS SERIES IN THEORETICAL AND APPLIED ECONOMICS 202016, University of Kansas, Department of Economics, revised Sep 2020.
    14. Tarassow, Artur, 2019. "Forecasting U.S. money growth using economic uncertainty measures and regularisation techniques," International Journal of Forecasting, Elsevier, vol. 35(2), pages 443-457.
    15. Allan Timmermann, 2018. "Forecasting Methods in Finance," Annual Review of Financial Economics, Annual Reviews, vol. 10(1), pages 449-479, November.
    16. Drachal, Krzysztof, 2021. "Forecasting selected energy commodities prices with Bayesian dynamic finite mixtures," Energy Economics, Elsevier, vol. 99(C).
    17. Krystian Jaworski, 2021. "Forecasting exchange rates for Central and Eastern European currencies using country‐specific factors," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 40(6), pages 977-999, September.
    18. Daniel Wochner, 2020. "Dynamic Factor Trees and Forests – A Theory-led Machine Learning Framework for Non-Linear and State-Dependent Short-Term U.S. GDP Growth Predictions," KOF Working papers 20-472, KOF Swiss Economic Institute, ETH Zurich.
    19. Cepni, Oguzhan & Güney, I. Ethem & Swanson, Norman R., 2019. "Nowcasting and forecasting GDP in emerging markets using global financial and macroeconomic diffusion indexes," International Journal of Forecasting, Elsevier, vol. 35(2), pages 555-572.
    20. Firat Melih Yilmaz & Ozer Arabaci, 2021. "Should Deep Learning Models be in High Demand, or Should They Simply be a Very Hot Topic? A Comprehensive Study for Exchange Rate Forecasting," Computational Economics, Springer;Society for Computational Economics, vol. 57(1), pages 217-245, January.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:


    Access and download statistics


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fgv:eesptd:498. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Núcleo de Computação da FGV EPGE (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.