IDEAS home Printed from https://ideas.repec.org/a/eee/empfin/v34y2015icp1-14.html
   My bibliography  Save this article

Significance testing in empirical finance: A critical review and assessment

Author

Listed:
  • Kim, Jae H.
  • Ji, Philip Inyeob

Abstract

This paper critically reviews the practice of significance testing in modern finance research. Employing a survey of recently published articles in four top-tier finance journals, we find that the conventional significance levels are exclusively used with little consideration of the key factors such as the sample size, power of the test, and expected losses. We also find that statistically significant results reported in many surveyed papers become questionable, if Bayesian method or revised standards for evidence were instead used. We observe strong evidence of publication bias in favour of statistical significance. We propose that substantial changes be made to the current practice of significance testing in finance research, in order to improve research credibility and integrity.

Suggested Citation

  • Kim, Jae H. & Ji, Philip Inyeob, 2015. "Significance testing in empirical finance: A critical review and assessment," Journal of Empirical Finance, Elsevier, vol. 34(C), pages 1-14.
    • Handle: RePEc:eee:empfin:v:34:y:2015:i:c:p:1-14
    DOI: 10.1016/j.jempfin.2015.08.006
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0927539815000894
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.jempfin.2015.08.006?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Deirdre N. McCloskey & Stephen T. Ziliak, 1996. "The Standard Error of Regressions," Journal of Economic Literature, American Economic Association, vol. 34(1), pages 97-114, March.
    2. Stephen T. Ziliak & Deirdre N. McCloskey, 2004. "Size Matters: The Standard Error of Regressions in the American Economic Review," Econ Journal Watch, Econ Journal Watch, vol. 1(2), pages 331-358, August.
    3. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    4. Paul D Ellis, 2010. "Effect sizes and the interpretation of research results in international business," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 41(9), pages 1581-1588, December.
    5. Keuzenkamp, Hugo A. & Magnus, Jan R., 1995. "On tests and significance in econometrics," Journal of Econometrics, Elsevier, vol. 67(1), pages 5-24, May.
    6. Clifford, Christopher P. & Jordan, Bradford D. & Riley, Timothy B., 2014. "Average funds versus average dollars: Implications for mutual fund research," Journal of Empirical Finance, Elsevier, vol. 28(C), pages 249-260.
    7. Morana, Claudio, 2014. "Insights on the global macro-finance interface: Structural sources of risk factor fluctuations and the cross-section of expected stock returns," Journal of Empirical Finance, Elsevier, vol. 29(C), pages 64-79.
    8. Kevin Hoover & Mark Siegler, 2008. "Sound and fury: McCloskey and significance testing in economics," Journal of Economic Methodology, Taylor & Francis Journals, vol. 15(1), pages 1-37.
    9. Contessi, Silvio & De Pace, Pierangelo & Guidolin, Massimo, 2014. "How did the financial crisis alter the correlations of U.S. yield spreads?," Journal of Empirical Finance, Elsevier, vol. 28(C), pages 362-385.
    10. Hassler, Uwe & Rodrigues, Paulo M.M. & Rubia, Antonio, 2014. "Persistence in the banking industry: Fractional integration and breaks in memory," Journal of Empirical Finance, Elsevier, vol. 29(C), pages 95-112.
    11. Soyer, Emre & Hogarth, Robin M., 2012. "The illusion of predictability: How regression statistics mislead experts," International Journal of Forecasting, Elsevier, vol. 28(3), pages 695-711.
    12. James G. MacKinnon, 2002. "Bootstrap inference in econometrics," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 35(4), pages 615-645, November.
    13. Sellke T. & Bayarri M. J. & Berger J. O., 2001. "Calibration of rho Values for Testing Precise Null Hypotheses," The American Statistician, American Statistical Association, vol. 55, pages 62-71, February.
    14. Mitchell A. Petersen, 2009. "Estimating Standard Errors in Finance Panel Data Sets: Comparing Approaches," The Review of Financial Studies, Society for Financial Studies, vol. 22(1), pages 435-480, January.
    15. Smith, Jason, 2014. "Does the market matter for more than investment?," Journal of Empirical Finance, Elsevier, vol. 25(C), pages 52-61.
    16. Sizova, Natalia, 2014. "A frequency-domain alternative to long-horizon regressions with application to return predictability," Journal of Empirical Finance, Elsevier, vol. 28(C), pages 261-272.
    17. De Long, J Bradford & Lang, Kevin, 1992. "Are All Economic Hypotheses False?," Journal of Political Economy, University of Chicago Press, vol. 100(6), pages 1257-1272, December.
    18. Kale, Jayant R. & Reis, Ebru & Venkateswaran, Anand, 2014. "Pay inequalities and managerial turnover," Journal of Empirical Finance, Elsevier, vol. 27(C), pages 21-39.
    19. Keef, Stephen P. & Khaled, Mohammed S., 2011. "Are investors moonstruck? Further international evidence on lunar phases and stock returns," Journal of Empirical Finance, Elsevier, vol. 18(1), pages 56-63, January.
    20. Connolly, Robert A., 1989. "An Examination of the Robustness of the Weekend Effect," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 24(2), pages 133-169, June.
    21. Baillie, Richard T. & Cho, Dooyeon, 2014. "Time variation in the standard forward premium regression: Some new models and tests," Journal of Empirical Finance, Elsevier, vol. 29(C), pages 52-63.
    22. Opschoor, Anne & Taylor, Nick & van der Wel, Michel & van Dijk, Dick, 2014. "Order flow and volatility: An empirical investigation," Journal of Empirical Finance, Elsevier, vol. 28(C), pages 185-201.
    23. Connolly, Robert A., 1991. "A posterior odds analysis of the weekend effect," Journal of Econometrics, Elsevier, vol. 49(1-2), pages 51-104.
    24. Rose, Annica, 2014. "The informational effect and market quality impact of upstairs trading and fleeting orders on the Australian Securities Exchange," Journal of Empirical Finance, Elsevier, vol. 28(C), pages 171-184.
    25. Thorbecke, Erik, 2004. "Economic and statistical significance: comments on "Size Matters"," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(5), pages 571-575, November.
    26. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    27. Tom Engsted, 2009. "Statistical vs. Economic Significance in Economics and Econometrics: Further comments on McCloskey & Ziliak," CREATES Research Papers 2009-17, Department of Economics and Business Economics, Aarhus University.
    28. Altman, Morris, 2004. "Statistical significance, path dependency, and the culture of journal publication," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(5), pages 651-663, November.
    29. Tom Engsted, 2009. "Statistical vs. economic significance in economics and econometrics: further comments on McCloskey and Ziliak," Journal of Economic Methodology, Taylor & Francis Journals, vol. 16(4), pages 393-408.
    30. Neal, Robert, 1987. "Potential Competition and Actual Competition in Equity Options," Journal of Finance, American Finance Association, vol. 42(3), pages 511-531, July.
    31. Donnelly, Catherine & Embrechts, Paul, 2010. "The Devil is in the Tails: Actuarial Mathematics and the Subprime Mortgage Crisis," ASTIN Bulletin, Cambridge University Press, vol. 40(1), pages 1-33, May.
    32. John Ioannidis & Chris Doucouliagos, 2013. "What'S To Know About The Credibility Of Empirical Economics?," Journal of Economic Surveys, Wiley Blackwell, vol. 27(5), pages 997-1004, December.
    33. Dahlquist, Magnus & Robertsson, Göran & Rydqvist, Kristian, 2014. "Direct evidence of dividend tax clienteles," Journal of Empirical Finance, Elsevier, vol. 28(C), pages 1-12.
    34. Klein, Roger W & Brown, Stephen J, 1984. "Model Selection When There Is "Minimal" Prior Information," Econometrica, Econometric Society, vol. 52(5), pages 1291-1312, September.
    35. Francis, Bill B. & Hasan, Iftekhar & Zhu, Yun, 2014. "Political uncertainty and bank loan contracting," Journal of Empirical Finance, Elsevier, vol. 29(C), pages 281-286.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kim, Jae H., 2017. "Stock returns and investors' mood: Good day sunshine or spurious correlation?," International Review of Financial Analysis, Elsevier, vol. 52(C), pages 94-103.
    2. Jae H. Kim & Kamran Ahmed & Philip Inyeob Ji, 2018. "Significance Testing in Accounting Research: A Critical Evaluation Based on Evidence," Abacus, Accounting Foundation, University of Sydney, vol. 54(4), pages 524-546, December.
    3. Alexander Libman & Joachim Zweynert, 2014. "Ceremonial Science: The State of Russian Economics Seen Through the Lens of the Work of ‘Doctor of Science’ Candidates," Working Papers 337, Leibniz Institut für Ost- und Südosteuropaforschung (Institute for East and Southeast European Studies).
    4. Libman, Alexander & Zweynert, Joachim, 2014. "Ceremonial science: The state of Russian economics seen through the lens of the work of ‘Doctor of Science’ candidates," Economic Systems, Elsevier, vol. 38(3), pages 360-378.
    5. Peter J. Veazie, 2015. "Understanding Statistical Testing," SAGE Open, , vol. 5(1), pages 21582440145, January.
    6. Thomas Mayer, 2012. "Ziliak and McCloskey's Criticisms of Significance Tests: An Assessment," Econ Journal Watch, Econ Journal Watch, vol. 9(3), pages 256-297, September.
    7. Thomas Mayer, 2012. "Ziliak and McClosky?s Criticisms of Significance Tests: A Damage Assessment," Working Papers 61, University of California, Davis, Department of Economics.
    8. Thomas Mayer, 2012. "Ziliak and McClosky?s Criticisms of Significance Tests: A Damage Assessment," Working Papers 126, University of California, Davis, Department of Economics.
    9. Jae H. Kim & In Choi, 2021. "Choosing the Level of Significance: A Decision‐theoretic Approach," Abacus, Accounting Foundation, University of Sydney, vol. 57(1), pages 27-71, March.
    10. Thomas Mayer, 2006. "The Empirical Significance of Econometric Models," Working Papers 620, University of California, Davis, Department of Economics.
    11. Black, Bernard & Hollingsworth, Alex & Nunes, Letícia & Simon, Kosali, 2022. "Simulated power analyses for observational studies: An application to the Affordable Care Act Medicaid expansion," Journal of Public Economics, Elsevier, vol. 213(C).
    12. Kim, Jae & Choi, In, 2015. "Unit Roots in Economic and Financial Time Series: A Re-Evaluation based on Enlightened Judgement," MPRA Paper 68411, University Library of Munich, Germany.
    13. Stephan B. Bruns & David I. Stern, 2019. "Lag length selection and p-hacking in Granger causality testing: prevalence and performance of meta-regression models," Empirical Economics, Springer, vol. 56(3), pages 797-830, March.
    14. Kim, Jae, 2015. "How to Choose the Level of Significance: A Pedagogical Note," MPRA Paper 66373, University Library of Munich, Germany.
    15. Hirschauer Norbert & Grüner Sven & Mußhoff Oliver & Becker Claudia, 2019. "Twenty Steps Towards an Adequate Inferential Interpretation of p-Values in Econometrics," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 239(4), pages 703-721, August.
    16. Meszaros, Sandor, 2008. "Theory testing (hypothesis testing) in agricultural economics," Studies in Agricultural Economics, Research Institute for Agricultural Economics, vol. 107, pages 1-13, March.
    17. Campbell R. Harvey & Yan Liu, 2020. "False (and Missed) Discoveries in Financial Economics," Papers 2006.04269, arXiv.org.
    18. Campbell R. Harvey & Yan Liu, 2020. "False (and Missed) Discoveries in Financial Economics," Journal of Finance, American Finance Association, vol. 75(5), pages 2503-2553, October.
    19. Christopher Snyder & Ran Zhuo, 2018. "Sniff Tests as a Screen in the Publication Process: Throwing out the Wheat with the Chaff," NBER Working Papers 25058, National Bureau of Economic Research, Inc.
    20. Stephen T. Ziliak & Deirdre N. McCloskey, 2013. "We Agree That Statistical Significance Proves Essentially Nothing: A Rejoinder to Thomas Mayer," Econ Journal Watch, Econ Journal Watch, vol. 10(1), pages 97-107, January.

    More about this item

    Keywords

    Level of significance; Lindley paradox; Massive sample size; Meehl's conjecture; Publication bias; Spurious statistical significance;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • G12 - Financial Economics - - General Financial Markets - - - Asset Pricing; Trading Volume; Bond Interest Rates

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:empfin:v:34:y:2015:i:c:p:1-14. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/jempfin .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.