IDEAS home Printed from https://ideas.repec.org/p/pra/mprapa/81664.html
   My bibliography  Save this paper

Significance Tests Harm Progress in Forecasting

Author

Listed:
  • Armstrong, J. Scott

Abstract

Based on a summary of prior literature, I conclude that tests of statistical significance harm scientific progress. Efforts to find exceptions to this conclusion have, to date, turned up none. Even when done correctly, significance tests are dangerous. I show that summaries of scientific research do not require tests of statistical significance. I illustrate the dangers of significance tests by examining an application to the M3-Competition. Although the authors of that reanalysis conducted a proper series of statistical tests, they suggest that the original M3 was not justified in concluding that combined forecasts reduce errors and that the selection of the best method is dependent upon the selection of a proper error measure. I show that the original conclusions were justified and that they are correct. Authors should try to avoid tests of statistical significance, journals should discourage them, and readers should ignore them. Instead, to analyze and communicate findings from empirical studies, one should use effect sizes, confidence intervals, replications/extensions, and meta-analyses.

Suggested Citation

  • Armstrong, J. Scott, 2007. "Significance Tests Harm Progress in Forecasting," MPRA Paper 81664, University Library of Munich, Germany.
    • Handle: RePEc:pra:mprapa:81664
    as

    Download full text from publisher

    File URL: https://mpra.ub.uni-muenchen.de/81664/1/MPRA_paper_81664.pdf
    File Function: original version
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Goodwin, Paul & Lawton, Richard, 2003. "Debiasing forecasts: how useful is the unbiasedness test?," International Journal of Forecasting, Elsevier, vol. 19(3), pages 467-475.
    2. Hubbard R. & Bayarri M.J., 2003. "Confusion Over Measures of Evidence (ps) Versus Errors (alphas) in Classical Statistical Testing," The American Statistician, American Statistical Association, vol. 57, pages 171-178, August.
    3. Armstrong, J. Scott, 2006. "Findings from evidence-based forecasting: Methods for reducing forecast error," International Journal of Forecasting, Elsevier, vol. 22(3), pages 583-598.
    4. Deirdre N. McCloskey & Stephen T. Ziliak, 1996. "The Standard Error of Regressions," Journal of Economic Literature, American Economic Association, vol. 34(1), pages 97-114, March.
    5. Makridakis, Spyros & Hibon, Michele, 2000. "The M3-Competition: results, conclusions and implications," International Journal of Forecasting, Elsevier, vol. 16(4), pages 451-476.
    6. Armstrong, J. Scott & Collopy, Fred, 1992. "Error measures for generalizing about forecasting methods: Empirical comparisons," International Journal of Forecasting, Elsevier, vol. 8(1), pages 69-80, June.
    7. Koning, Alex J. & Franses, Philip Hans & Hibon, Michele & Stekler, H.O., 2005. "The M3 competition: Statistical tests of the results," International Journal of Forecasting, Elsevier, vol. 21(3), pages 397-409.
    8. Wright, Malcolm & Armstrong, J. Scott, 2007. "Verification of Citations: Fawlty Towers of Knowledge?," MPRA Paper 4149, University Library of Munich, Germany.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Armstrong, J. Scott & Green, Kesten C. & Graefe, Andreas, 2015. "Golden rule of forecasting: Be conservative," Journal of Business Research, Elsevier, vol. 68(8), pages 1717-1731.
    2. Crone, Sven F. & Hibon, Michèle & Nikolopoulos, Konstantinos, 2011. "Advances in forecasting with neural networks? Empirical evidence from the NN3 competition on time series prediction," International Journal of Forecasting, Elsevier, vol. 27(3), pages 635-660, July.
    3. Schaer, Oliver & Kourentzes, Nikolaos & Fildes, Robert, 2019. "Demand forecasting with user-generated online information," International Journal of Forecasting, Elsevier, vol. 35(1), pages 197-212.
    4. Hyndman, Rob J. & Koehler, Anne B., 2006. "Another look at measures of forecast accuracy," International Journal of Forecasting, Elsevier, vol. 22(4), pages 679-688.
    5. Fildes, Robert, 2006. "The forecasting journals and their contribution to forecasting research: Citation analysis and expert opinion," International Journal of Forecasting, Elsevier, vol. 22(3), pages 415-432.
    6. Hill, Arthur V. & Zhang, Weiyong & Burch, Gerald F., 2015. "Forecasting the forecastability quotient for inventory management," International Journal of Forecasting, Elsevier, vol. 31(3), pages 651-663.
    7. Makridakis, Spyros & Spiliotis, Evangelos & Assimakopoulos, Vassilios, 2020. "The M4 Competition: 100,000 time series and 61 forecasting methods," International Journal of Forecasting, Elsevier, vol. 36(1), pages 54-74.
    8. Theodosiou, Marina, 2011. "Forecasting monthly and quarterly time series using STL decomposition," International Journal of Forecasting, Elsevier, vol. 27(4), pages 1178-1195, October.
    9. Kourentzes, Nikolaos & Petropoulos, Fotios & Trapero, Juan R., 2014. "Improving forecasting by estimating time series structural components across multiple frequencies," International Journal of Forecasting, Elsevier, vol. 30(2), pages 291-302.
    10. Meira, Erick & Cyrino Oliveira, Fernando Luiz & de Menezes, Lilian M., 2022. "Forecasting natural gas consumption using Bagging and modified regularization techniques," Energy Economics, Elsevier, vol. 106(C).
    11. Dean W. Wichern & Benito E. Flores, 2005. "Evaluating forecasts: a look at aggregate bias and accuracy measures," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 24(6), pages 433-451.
    12. Makridakis, Spyros & Hyndman, Rob J. & Petropoulos, Fotios, 2020. "Forecasting in social settings: The state of the art," International Journal of Forecasting, Elsevier, vol. 36(1), pages 15-28.
    13. Davide Provenzano & Serena Volo, 2022. "Tourism recovery amid COVID-19: The case of Lombardy, Italy," Tourism Economics, , vol. 28(1), pages 110-130, February.
    14. Lawrence, Michael & O'Connor, Marcus, 2005. "Judgmental forecasting in the presence of loss functions," International Journal of Forecasting, Elsevier, vol. 21(1), pages 3-14.
    15. Villegas, Marco A. & Pedregal, Diego J., 2019. "Automatic selection of unobserved components models for supply chain forecasting," International Journal of Forecasting, Elsevier, vol. 35(1), pages 157-169.
    16. Armstrong, J. Scott & Brodie, Roderick J., 1999. "Forecasting for Marketing," MPRA Paper 81690, University Library of Munich, Germany.
    17. Gary Madden & Joachim Tan, 2008. "Forecasting international bandwidth capacity using linear and ANN methods," Applied Economics, Taylor & Francis Journals, vol. 40(14), pages 1775-1787.
    18. Ji Wu & Xian Cheng & Stephen Shaoyi Liao, 2020. "Tourism forecast combination using the stochastic frontier analysis technique," Tourism Economics, , vol. 26(7), pages 1086-1107, November.
    19. Robert R. Andrawis & Amir F. Atiya, 2009. "A new Bayesian formulation for Holt's exponential smoothing," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 28(3), pages 218-234.
    20. Jennifer L. Castle & Jurgen A. Doornik & David F. Hendry, 2021. "Forecasting Principles from Experience with Forecasting Competitions," Forecasting, MDPI, vol. 3(1), pages 1-28, February.

    More about this item

    Keywords

    accuracy measures; combining forecasts; confidence intervals; effect size; M-competition; meta-analysis; null hypothesis; practical significance; replications;
    All these keywords.

    JEL classification:

    • C1 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pra:mprapa:81664. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Joachim Winter (email available below). General contact details of provider: https://edirc.repec.org/data/vfmunde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.