IDEAS home Printed from https://ideas.repec.org/a/eee/empfin/v34y2015icp1-14.html
   My bibliography  Save this article

Significance testing in empirical finance: A critical review and assessment

Author

Listed:
  • Kim, Jae H.
  • Ji, Philip Inyeob

Abstract

This paper critically reviews the practice of significance testing in modern finance research. Employing a survey of recently published articles in four top-tier finance journals, we find that the conventional significance levels are exclusively used with little consideration of the key factors such as the sample size, power of the test, and expected losses. We also find that statistically significant results reported in many surveyed papers become questionable, if Bayesian method or revised standards for evidence were instead used. We observe strong evidence of publication bias in favour of statistical significance. We propose that substantial changes be made to the current practice of significance testing in finance research, in order to improve research credibility and integrity.

Suggested Citation

  • Kim, Jae H. & Ji, Philip Inyeob, 2015. "Significance testing in empirical finance: A critical review and assessment," Journal of Empirical Finance, Elsevier, vol. 34(C), pages 1-14.
  • Handle: RePEc:eee:empfin:v:34:y:2015:i:c:p:1-14
    DOI: 10.1016/j.jempfin.2015.08.006
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0927539815000894
    Download Restriction: Full text for ScienceDirect subscribers only

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Deirdre N. McCloskey & Stephen T. Ziliak, 1996. "The Standard Error of Regressions," Journal of Economic Literature, American Economic Association, vol. 34(1), pages 97-114, March.
    2. Stephen T. Ziliak & Deirdre N. McCloskey, 2004. "Size Matters: The Standard Error of Regressions in the American Economic Review," Econ Journal Watch, Econ Journal Watch, vol. 1(2), pages 331-358, August.
    3. Paul D Ellis, 2010. "Effect sizes and the interpretation of research results in international business," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 41(9), pages 1581-1588, December.
    4. Keuzenkamp, H.A. & Magnus, J.R., 1995. "On tests and significance in econometrics," Other publications TiSEM 1808e2e0-3805-4999-b9a1-5, Tilburg University, School of Economics and Management.
    5. Clifford, Christopher P. & Jordan, Bradford D. & Riley, Timothy B., 2014. "Average funds versus average dollars: Implications for mutual fund research," Journal of Empirical Finance, Elsevier, vol. 28(C), pages 249-260.
    6. Morana, Claudio, 2014. "Insights on the global macro-finance interface: Structural sources of risk factor fluctuations and the cross-section of expected stock returns," Journal of Empirical Finance, Elsevier, vol. 29(C), pages 64-79.
    7. Kevin Hoover & Mark Siegler, 2008. "Sound and fury: McCloskey and significance testing in economics," Journal of Economic Methodology, Taylor & Francis Journals, vol. 15(1), pages 1-37.
    8. Contessi, Silvio & De Pace, Pierangelo & Guidolin, Massimo, 2014. "How did the financial crisis alter the correlations of U.S. yield spreads?," Journal of Empirical Finance, Elsevier, vol. 28(C), pages 362-385.
    9. Hassler, Uwe & Rodrigues, Paulo M.M. & Rubia, Antonio, 2014. "Persistence in the banking industry: Fractional integration and breaks in memory," Journal of Empirical Finance, Elsevier, vol. 29(C), pages 95-112.
    10. Soyer, Emre & Hogarth, Robin M., 2012. "The illusion of predictability: How regression statistics mislead experts," International Journal of Forecasting, Elsevier, vol. 28(3), pages 695-711.
    11. James G. MacKinnon, 2002. "Bootstrap inference in econometrics," Canadian Journal of Economics, Canadian Economics Association, vol. 35(4), pages 615-645, November.
    12. Mitchell A. Petersen, 2009. "Estimating Standard Errors in Finance Panel Data Sets: Comparing Approaches," Review of Financial Studies, Society for Financial Studies, vol. 22(1), pages 435-480, January.
    13. Keuzenkamp, Hugo A. & Magnus, Jan R., 1995. "On tests and significance in econometrics," Journal of Econometrics, Elsevier, vol. 67(1), pages 5-24, May.
    14. Smith, Jason, 2014. "Does the market matter for more than investment?," Journal of Empirical Finance, Elsevier, vol. 25(C), pages 52-61.
    15. Sizova, Natalia, 2014. "A frequency-domain alternative to long-horizon regressions with application to return predictability," Journal of Empirical Finance, Elsevier, vol. 28(C), pages 261-272.
    16. De Long, J Bradford & Lang, Kevin, 1992. "Are All Economic Hypotheses False?," Journal of Political Economy, University of Chicago Press, vol. 100(6), pages 1257-1272, December.
    17. Kale, Jayant R. & Reis, Ebru & Venkateswaran, Anand, 2014. "Pay inequalities and managerial turnover," Journal of Empirical Finance, Elsevier, vol. 27(C), pages 21-39.
    18. Keef, Stephen P. & Khaled, Mohammed S., 2011. "Are investors moonstruck? Further international evidence on lunar phases and stock returns," Journal of Empirical Finance, Elsevier, vol. 18(1), pages 56-63, January.
    19. Connolly, Robert A., 1989. "An Examination of the Robustness of the Weekend Effect," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 24(2), pages 133-169, June.
    20. Baillie, Richard T. & Cho, Dooyeon, 2014. "Time variation in the standard forward premium regression: Some new models and tests," Journal of Empirical Finance, Elsevier, vol. 29(C), pages 52-63.
    21. Opschoor, Anne & Taylor, Nick & van der Wel, Michel & van Dijk, Dick, 2014. "Order flow and volatility: An empirical investigation," Journal of Empirical Finance, Elsevier, vol. 28(C), pages 185-201.
    22. Connolly, Robert A., 1991. "A posterior odds analysis of the weekend effect," Journal of Econometrics, Elsevier, vol. 49(1-2), pages 51-104.
    23. Rose, Annica, 2014. "The informational effect and market quality impact of upstairs trading and fleeting orders on the Australian Securities Exchange," Journal of Empirical Finance, Elsevier, vol. 28(C), pages 171-184.
    24. Thorbecke, Erik, 2004. "Economic and statistical significance: comments on "Size Matters"," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(5), pages 571-575, November.
    25. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    26. Tom Engsted, 2009. "Statistical vs. Economic Significance in Economics and Econometrics: Further comments on McCloskey & Ziliak," CREATES Research Papers 2009-17, Department of Economics and Business Economics, Aarhus University.
    27. Altman, Morris, 2004. "Statistical significance, path dependency, and the culture of journal publication," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(5), pages 651-663, November.
    28. Tom Engsted, 2009. "Statistical vs. economic significance in economics and econometrics: further comments on McCloskey and Ziliak," Journal of Economic Methodology, Taylor & Francis Journals, vol. 16(4), pages 393-408.
    29. Neal, Robert, 1987. " Potential Competition and Actual Competition in Equity Options," Journal of Finance, American Finance Association, vol. 42(3), pages 511-531, July.
    30. Donnelly, Catherine & Embrechts, Paul, 2010. "The Devil is in the Tails: Actuarial Mathematics and the Subprime Mortgage Crisis," ASTIN Bulletin, Cambridge University Press, vol. 40(1), pages 1-33, May.
    31. John Ioannidis & Chris Doucouliagos, 2013. "What'S To Know About The Credibility Of Empirical Economics?," Journal of Economic Surveys, Wiley Blackwell, vol. 27(5), pages 997-1004, December.
    32. Dahlquist, Magnus & Robertsson, Göran & Rydqvist, Kristian, 2014. "Direct evidence of dividend tax clienteles," Journal of Empirical Finance, Elsevier, vol. 28(C), pages 1-12.
    33. Klein, Roger W & Brown, Stephen J, 1984. "Model Selection When There Is "Minimal" Prior Information," Econometrica, Econometric Society, vol. 52(5), pages 1291-1312, September.
    34. Francis, Bill B. & Hasan, Iftekhar & Zhu, Yun, 2014. "Political uncertainty and bank loan contracting," Journal of Empirical Finance, Elsevier, vol. 29(C), pages 281-286.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kim, Jae H., 2017. "Stock returns and investors' mood: Good day sunshine or spurious correlation?," International Review of Financial Analysis, Elsevier, vol. 52(C), pages 94-103.
    2. Stephan B. Bruns & David I. Stern, 2019. "Lag length selection and p-hacking in Granger causality testing: prevalence and performance of meta-regression models," Empirical Economics, Springer, vol. 56(3), pages 797-830, March.
    3. Jae H. Kim & Kamran Ahmed & Philip Inyeob Ji, 2018. "Significance Testing in Accounting Research: A Critical Evaluation Based on Evidence," Abacus, Accounting Foundation, University of Sydney, vol. 54(4), pages 524-546, December.
    4. Yergeau, Gabriel, 2016. "Profitability and Market Quality of High Frequency Market-makers: An Empirical Investigation," Working Papers 16-3, HEC Montreal, Canada Research Chair in Risk Management.
    5. Jae H. Kim & Andrew P. Robinson, 2019. "Interval-Based Hypothesis Testing and Its Applications to Economics and Finance," Econometrics, MDPI, Open Access Journal, vol. 7(2), pages 1-22, May.
    6. David Trafimow, 2019. "A Frequentist Alternative to Significance Testing, p -Values, and Confidence Intervals," Econometrics, MDPI, Open Access Journal, vol. 7(2), pages 1-14, June.
    7. Jae H. Kim & In Choi, 2017. "Unit Roots in Economic and Financial Time Series: A Re-Evaluation at the Decision-Based Significance Levels," Econometrics, MDPI, Open Access Journal, vol. 5(3), pages 1-23, September.
    8. Kim, Jae, 2015. "How to Choose the Level of Significance: A Pedagogical Note," MPRA Paper 66373, University Library of Munich, Germany.
    9. Kim, Jae & Choi, In, 2015. "Unit Roots in Economic and Financial Time Series: A Re-Evaluation based on Enlightened Judgement," MPRA Paper 68411, University Library of Munich, Germany.
    10. Richard Startz, 2019. "Not p -Values, Said a Little Bit Differently," Econometrics, MDPI, Open Access Journal, vol. 7(1), pages 1-5, March.

    More about this item

    Keywords

    Level of significance; Lindley paradox; Massive sample size; Meehl's conjecture; Publication bias; Spurious statistical significance;

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • G12 - Financial Economics - - General Financial Markets - - - Asset Pricing; Trading Volume; Bond Interest Rates

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:empfin:v:34:y:2015:i:c:p:1-14. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Dana Niculescu). General contact details of provider: http://www.elsevier.com/locate/jempfin .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.