IDEAS home Printed from https://ideas.repec.org/a/bla/abacus/v54y2018i4p524-546.html
   My bibliography  Save this article

Significance Testing in Accounting Research: A Critical Evaluation Based on Evidence

Author

Listed:
  • Jae H. Kim
  • Kamran Ahmed
  • Philip Inyeob Ji

Abstract

From a survey of the papers published in leading accounting journals in 2014, we find that accounting researchers conduct significance testing almost exclusively at a conventional level of significance, without considering key factors such as the sample size or power of a test. We present evidence that a vast majority of the accounting studies favour large or massive sample sizes and conduct significance tests with the power extremely close to or equal to one. As a result, statistical inference is severely biased towards Type I error, frequently rejecting the true null hypotheses. Under the ‘p‐value less than 0.05’ criterion for statistical significance, more than 90% of the surveyed papers report statistical significance. However, under alternative criteria, only 40% of the results are statistically significant. We propose that substantial changes be made to the current practice of significance testing for more credible empirical research in accounting.

Suggested Citation

  • Jae H. Kim & Kamran Ahmed & Philip Inyeob Ji, 2018. "Significance Testing in Accounting Research: A Critical Evaluation Based on Evidence," Abacus, Accounting Foundation, University of Sydney, vol. 54(4), pages 524-546, December.
    • Handle: RePEc:bla:abacus:v:54:y:2018:i:4:p:524-546
    DOI: 10.1111/abac.12141
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/abac.12141
    Download Restriction: no
    File URL: https://libkey.io/10.1111/abac.12141?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Deirdre N. McCloskey & Stephen T. Ziliak, 1996. "The Standard Error of Regressions," Journal of Economic Literature, American Economic Association, vol. 34(1), pages 97-114, March.
    2. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    3. Startz, Richard, 2014. "Choosing the More Likely Hypothesis," Foundations and Trends(R) in Econometrics, now publishers, vol. 7(2), pages 119-189, November.
    4. Keuzenkamp, Hugo A. & Magnus, Jan R., 1995. "On tests and significance in econometrics," Journal of Econometrics, Elsevier, vol. 67(1), pages 5-24, May.
    5. David Hirshleifer & Tyler Shumway, 2003. "Good Day Sunshine: Stock Returns and the Weather," Journal of Finance, American Finance Association, vol. 58(3), pages 1009-1032, June.
    6. Davidson, Russell & MacKinnon, James G., 1993. "Estimation and Inference in Econometrics," OUP Catalogue, Oxford University Press, number 9780195060119.
    7. Kim, Jae H. & Ji, Philip Inyeob, 2015. "Significance testing in empirical finance: A critical review and assessment," Journal of Empirical Finance, Elsevier, vol. 34(C), pages 1-14.
    8. Grossman, Sanford J & Stiglitz, Joseph E, 1980. "On the Impossibility of Informationally Efficient Markets," American Economic Review, American Economic Association, vol. 70(3), pages 393-408, June.
    9. R. Murray Lindsay, 1994. "Publication System Biases Associated with the Statistical Testing Paradigm," Contemporary Accounting Research, John Wiley & Sons, vol. 11(1), pages 33-57, June.
    10. Thomas R. Dyckman, 2016. "Significance Testing: We Can Do Better," Abacus, Accounting Foundation, University of Sydney, vol. 52(2), pages 319-342, June.
    11. De Long, J Bradford & Lang, Kevin, 1992. "Are All Economic Hypotheses False?," Journal of Political Economy, University of Chicago Press, vol. 100(6), pages 1257-1272, December.
    12. Kim, Jae H., 2017. "Stock returns and investors' mood: Good day sunshine or spurious correlation?," International Review of Financial Analysis, Elsevier, vol. 52(C), pages 94-103.
    13. Lindsay, R. Murray, 1995. "Reconsidering the status of tests of significance: An alternative criterion of adequacy," Accounting, Organizations and Society, Elsevier, vol. 20(1), pages 35-53, January.
    14. Mitchell A. Petersen, 2009. "Estimating Standard Errors in Finance Panel Data Sets: Comparing Approaches," The Review of Financial Studies, Society for Financial Studies, vol. 22(1), pages 435-480, January.
    15. Tom Engsted, 2009. "Statistical vs. Economic Significance in Economics and Econometrics: Further comments on McCloskey & Ziliak," CREATES Research Papers 2009-17, Department of Economics and Business Economics, Aarhus University.
    16. Steven E. Salterio, 2014. "We Don't Replicate Accounting Research—Or Do We?," Contemporary Accounting Research, John Wiley & Sons, vol. 31(4), pages 1134-1142, December.
    17. Jae H. Kim & In Choi, 2017. "Unit Roots in Economic and Financial Time Series: A Re-Evaluation at the Decision-Based Significance Levels," Econometrics, MDPI, vol. 5(3), pages 1-23, September.
    18. Imad A. Moosa, 2017. "Econometrics as a Con Art," Books, Edward Elgar Publishing, number 17257.
    19. James A. Ohlson, 2015. "Accounting Research and Common Sense," Abacus, Accounting Foundation, University of Sydney, vol. 51(4), pages 525-535, December.
    20. Tom Engsted, 2009. "Statistical vs. economic significance in economics and econometrics: further comments on McCloskey and Ziliak," Journal of Economic Methodology, Taylor & Francis Journals, vol. 16(4), pages 393-408.
    21. Dale J. Poirier, 1995. "Intermediate Statistics and Econometrics: A Comparative Approach," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262161494, December.
    22. Campbell R. Harvey, 2017. "Presidential Address: The Scientific Outlook in Financial Economics," Journal of Finance, American Finance Association, vol. 72(4), pages 1399-1440, August.
    23. John Ioannidis & Chris Doucouliagos, 2013. "What'S To Know About The Credibility Of Empirical Economics?," Journal of Economic Surveys, Wiley Blackwell, vol. 27(5), pages 997-1004, December.
    24. Pérez, María-Eglée & Pericchi, Luis Raúl, 2014. "Changing statistical significance with the amount of information: The adaptive α significance level," Statistics & Probability Letters, Elsevier, vol. 85(C), pages 20-24.
    25. R. Murray Lindsay, 1997. "Lies, Damned Lies and More Statistics: The Neglected Issue of Multiplicity in Accounting Research," Accounting and Business Research, Taylor & Francis Journals, vol. 27(3), pages 243-258, February.
    26. Mingfeng Lin & Henry C. Lucas & Galit Shmueli, 2013. "Research Commentary ---Too Big to Fail: Large Samples and the p -Value Problem," Information Systems Research, INFORMS, vol. 24(4), pages 906-917, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Muhammad Ishaq Bhatti & Jae H. Kim, 2020. "Towards a New Paradigm for Statistical Evidence in the Use of p -Value," Econometrics, MDPI, vol. 9(1), pages 1-3, December.
    2. Jae H. Kim & Andrew P. Robinson, 2019. "Interval-Based Hypothesis Testing and Its Applications to Economics and Finance," Econometrics, MDPI, vol. 7(2), pages 1-22, May.
    3. Xi Fu & Xiaoxi Wu & Zhifang Zhang, 2021. "The Information Role of Earnings Conference Call Tone: Evidence from Stock Price Crash Risk," Journal of Business Ethics, Springer, vol. 173(3), pages 643-660, October.
    4. Jones, Stewart & Wang, Tim, 2019. "Predicting private company failure: A multi-class analysis," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 61(C), pages 161-188.
    5. Kellner, Ralf & Rösch, Daniel, 2021. "A Bayesian Re-Interpretation of “significant” empirical financial research," Finance Research Letters, Elsevier, vol. 38(C).
    6. Dan Hu & Eunju Lee & Bingxin Li, 2023. "Trade secrets protection and stock price crash risk," The Financial Review, Eastern Finance Association, vol. 58(2), pages 395-421, May.
    7. Johnstone, David, 2022. "Accounting research and the significance test crisis," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 89(C).
    8. Thomas R. Dyckman & Stephen A. Zeff, 2019. "Important Issues in Statistical Testing and Recommended Improvements in Accounting Research," Econometrics, MDPI, vol. 7(2), pages 1-11, May.
    9. Chen, Xiaomeng Charlene & Jones, Stewart & Hasan, Mostafa Monzur & Zhao, Ruoyun & Alam, Nurul, 2023. "Does strategic deviation influence firms’ use of supplier finance?," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 85(C).
    10. Jae H. Kim & In Choi, 2021. "Choosing the Level of Significance: A Decision‐theoretic Approach," Abacus, Accounting Foundation, University of Sydney, vol. 57(1), pages 27-71, March.
    11. James A. Ohlson, 2022. "Researchers’ data analysis choices: an excess of false positives?," Review of Accounting Studies, Springer, vol. 27(2), pages 649-667, June.
    12. Stewart Jones & Nurul Alam, 2019. "A machine learning analysis of citation impact among selected Pacific Basin journals," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 59(4), pages 2509-2552, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jae H. Kim & Andrew P. Robinson, 2019. "Interval-Based Hypothesis Testing and Its Applications to Economics and Finance," Econometrics, MDPI, vol. 7(2), pages 1-22, May.
    2. Jae H. Kim & In Choi, 2021. "Choosing the Level of Significance: A Decision‐theoretic Approach," Abacus, Accounting Foundation, University of Sydney, vol. 57(1), pages 27-71, March.
    3. Jae H. Kim, 2022. "Moving to a world beyond p-value," Review of Managerial Science, Springer, vol. 16(8), pages 2467-2493, November.
    4. Kim, Jae H., 2017. "Stock returns and investors' mood: Good day sunshine or spurious correlation?," International Review of Financial Analysis, Elsevier, vol. 52(C), pages 94-103.
    5. Kim, Jae H. & Ji, Philip Inyeob, 2015. "Significance testing in empirical finance: A critical review and assessment," Journal of Empirical Finance, Elsevier, vol. 34(C), pages 1-14.
    6. Kim, Jae & Choi, In, 2015. "Unit Roots in Economic and Financial Time Series: A Re-Evaluation based on Enlightened Judgement," MPRA Paper 68411, University Library of Munich, Germany.
    7. Johnstone, David, 2022. "Accounting research and the significance test crisis," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 89(C).
    8. Michaelides, Michael, 2021. "Large sample size bias in empirical finance," Finance Research Letters, Elsevier, vol. 41(C).
    9. Liu, Huajin & Zhang, Wei & Zhang, Xiaotao & Liu, Jia, 2021. "Temperature and trading behaviours," International Review of Financial Analysis, Elsevier, vol. 78(C).
    10. Thomas R. Dyckman & Stephen A. Zeff, 2019. "Important Issues in Statistical Testing and Recommended Improvements in Accounting Research," Econometrics, MDPI, vol. 7(2), pages 1-11, May.
    11. Todd Mitton, 2022. "Methodological Variation in Empirical Corporate Finance," The Review of Financial Studies, Society for Financial Studies, vol. 35(2), pages 527-575.
    12. Stephan B. Bruns & David I. Stern, 2019. "Lag length selection and p-hacking in Granger causality testing: prevalence and performance of meta-regression models," Empirical Economics, Springer, vol. 56(3), pages 797-830, March.
    13. Muhammad Ishaq Bhatti & Jae H. Kim, 2020. "Towards a New Paradigm for Statistical Evidence in the Use of p -Value," Econometrics, MDPI, vol. 9(1), pages 1-3, December.
    14. Jae H. Kim & In Choi, 2017. "Unit Roots in Economic and Financial Time Series: A Re-Evaluation at the Decision-Based Significance Levels," Econometrics, MDPI, vol. 5(3), pages 1-23, September.
    15. Kim, Jae, 2015. "How to Choose the Level of Significance: A Pedagogical Note," MPRA Paper 66373, University Library of Munich, Germany.
    16. Alexander Libman & Joachim Zweynert, 2014. "Ceremonial Science: The State of Russian Economics Seen Through the Lens of the Work of ‘Doctor of Science’ Candidates," Working Papers 337, Leibniz Institut für Ost- und Südosteuropaforschung (Institute for East and Southeast European Studies).
    17. Libman, Alexander & Zweynert, Joachim, 2014. "Ceremonial science: The state of Russian economics seen through the lens of the work of ‘Doctor of Science’ candidates," Economic Systems, Elsevier, vol. 38(3), pages 360-378.
    18. Jeremy Arkes, 2020. "Teaching Graduate (and Undergraduate) Econometrics: Some Sensible Shifts to Improve Efficiency, Effectiveness, and Usefulness," Econometrics, MDPI, vol. 8(3), pages 1-23, September.
    19. Campbell R. Harvey & Yan Liu, 2020. "False (and Missed) Discoveries in Financial Economics," Papers 2006.04269, arXiv.org.
    20. Kellner, Ralf & Rösch, Daniel, 2021. "A Bayesian Re-Interpretation of “significant” empirical financial research," Finance Research Letters, Elsevier, vol. 38(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:abacus:v:54:y:2018:i:4:p:524-546. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.blackwellpublishing.com/journal.asp?ref=0001-3072 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.