IDEAS home Printed from https://ideas.repec.org/a/eee/crpeac/v89y2022ics1045235421000150.html
   My bibliography  Save this article

Accounting research and the significance test crisis

Author

Listed:
  • Johnstone, David

Abstract

The emerging or at least threatening “significance test crisis” in accounting has been prompted by a chorus across multiple physical and social sciences of dissatisfaction with conventional frequentist statistical research methods and behaviors, particularly the use and abuse of p-levels. There are now hundreds of published papers and statements, echoing what has been said behind closed doors for decades, namely that much if not most empirical research is unreliable, simply wrong or at worst fabricated. The problems are a mixture of flawed statistical logic (as Bayesians have claimed for decades), “p-hacking” by way of fishing for significant results and publications, selective reporting or “the file drawer problem”, and ultimately the “agency problem” that researchers charged by funding bodies (their Universities, governments and taxpayers) with conducting disinterested “objective science” are motivated more by the personal need to publish and please other researchers. Expanding on that theme, the supply of empirical research in the “market for statistical significance” is described in terms of “market failure” and “the market for lemons”.

Suggested Citation

  • Johnstone, David, 2022. "Accounting research and the significance test crisis," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 89(C).
  • Handle: RePEc:eee:crpeac:v:89:y:2022:i:c:s1045235421000150
    DOI: 10.1016/j.cpa.2021.102296
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1045235421000150
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.cpa.2021.102296?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. John Quiggin, 2019. "The Replication Crisis as Market Failure," Econometrics, MDPI, vol. 7(4), pages 1-8, November.
    2. Anthony Hopwood, 2008. "Changing Pressures on the Research Process: On Trying to Research in an Age when Curiosity is not Enough," European Accounting Review, Taylor & Francis Journals, vol. 17(1), pages 87-96.
    3. John Dumay & Charl de Villiers, 2019. "Qualitative accounting research: special issue introduction," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 59(3), pages 1449-1458, September.
    4. Daniel J. Benjamin & James O. Berger & Magnus Johannesson & Brian A. Nosek & E.-J. Wagenmakers & Richard Berk & Kenneth A. Bollen & Björn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Chr, 2018. "Redefine statistical significance," Nature Human Behaviour, Nature, vol. 2(1), pages 6-10, January.
      • Daniel Benjamin & James Berger & Magnus Johannesson & Brian Nosek & E. Wagenmakers & Richard Berk & Kenneth Bollen & Bjorn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Christopher Chambe, 2017. "Redefine Statistical Significance," Artefactual Field Experiments 00612, The Field Experiments Website.
    5. Michael Muthukrishna & Joseph Henrich, 2019. "A problem in theory," Nature Human Behaviour, Nature, vol. 3(3), pages 221-229, March.
    6. Chen, Qi & Schipper, Katherine, 2016. "Comments and Observations Regarding the Relation Between Theory and Empirical Research in Contemporary Accounting Research," Foundations and Trends(R) in Accounting, now publishers, vol. 10(2-4), pages 314-360, August.
    7. Regina Nuzzo, 2014. "Scientific method: Statistical errors," Nature, Nature, vol. 506(7487), pages 150-152, February.
    8. Stephan B Bruns & John P A Ioannidis, 2016. "p-Curve and p-Hacking in Observational Research," PLOS ONE, Public Library of Science, vol. 11(2), pages 1-13, February.
    9. Ravenscroft, Sue & Williams, Paul F., 2009. "Making imaginary worlds real: The case of expensing employee stock options," Accounting, Organizations and Society, Elsevier, vol. 34(6-7), pages 770-786, August.
    10. Harvey, Campbell R. & Zhou, Guofu, 1990. "Bayesian inference in asset pricing tests," Journal of Financial Economics, Elsevier, vol. 26(2), pages 221-254, August.
    11. Blakeley B. McShane & David Gal & Andrew Gelman & Christian Robert & Jennifer L. Tackett, 2019. "Abandon Statistical Significance," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 235-245, March.
    12. Campbell R. Harvey, 2017. "Presidential Address: The Scientific Outlook in Financial Economics," Journal of Finance, American Finance Association, vol. 72(4), pages 1399-1440, August.
    13. Verrecchia, Robert E., 2001. "Essays on disclosure," Journal of Accounting and Economics, Elsevier, vol. 32(1-3), pages 97-180, December.
    14. Jennifer Francis & Ryan Lafond & Per Olsson & Katherine Schipper, 2007. "Information Uncertainty and Post-Earnings-Announcement-Drift," Journal of Business Finance & Accounting, Wiley Blackwell, vol. 34(3-4), pages 403-433.
    15. Dewald, William G & Thursby, Jerry G & Anderson, Richard G, 1986. "Replication in Empirical Economics: The Journal of Money, Credit and Banking Project," American Economic Review, American Economic Association, vol. 76(4), pages 587-603, September.
    16. Bäuerle Nicole & Chen An, 2019. "Optimal retirement planning under partial information," Statistics & Risk Modeling, De Gruyter, vol. 36(1-4), pages 37-55, December.
    17. Thomas R. Dyckman & Stephen A. Zeff, 2019. "Important Issues in Statistical Testing and Recommended Improvements in Accounting Research," Econometrics, MDPI, vol. 7(2), pages 1-11, May.
    18. Williams, Paul F., 1989. "The logic of positive accounting research," Accounting, Organizations and Society, Elsevier, vol. 14(5-6), pages 455-468, October.
    19. Ball, R & Brown, P, 1968. "Empirical Evaluation Of Accounting Income Numbers," Journal of Accounting Research, Wiley Blackwell, vol. 6(2), pages 159-178.
    20. Charles N. Bailey & James R. Hasselback & Julia N. Karcher, 2001. "Research Misconduct in Accounting Literature: A Survey of the Most Prolific Researchers’ Actions and Beliefs," Abacus, Accounting Foundation, University of Sydney, vol. 37(1), pages 26-54, February.
    21. Ronald L. Wasserstein & Nicole A. Lazar, 2016. "The ASA's Statement on p -Values: Context, Process, and Purpose," The American Statistician, Taylor & Francis Journals, vol. 70(2), pages 129-133, May.
    22. Christian Leuz, 2018. "Evidence-based policymaking: promise, challenges and opportunities for accounting and financial markets research," Accounting and Business Research, Taylor & Francis Journals, vol. 48(5), pages 582-608, July.
    23. Harvey, Campbell R., 2019. "Editorial: Replication in Financial Economics," Critical Finance Review, now publishers, vol. 8(1-2), pages 1-9, December.
    24. Valentin Amrhein & Sander Greenland & Blake McShane, 2019. "Scientists rise up against statistical significance," Nature, Nature, vol. 567(7748), pages 305-307, March.
    25. Nolan, Anne & Whelan, Adele & McGuinness, Seamus & Maître, Bertrand, 2019. "Gender, pensions and income in retirement," Research Series, Economic and Social Research Institute (ESRI), number RS87, June.
    26. Andreas G. F. Hoepner & David McMillan & Andrew Vivian & Chardin Wese Simen, 2021. "Significance, relevance and explainability in the machine learning age: an econometrics and financial data science perspective," The European Journal of Finance, Taylor & Francis Journals, vol. 27(1-2), pages 1-7, January.
    27. Kim, Jae H. & Ji, Philip Inyeob, 2015. "Significance testing in empirical finance: A critical review and assessment," Journal of Empirical Finance, Elsevier, vol. 34(C), pages 1-14.
    28. R. Murray Lindsay, 1994. "Publication System Biases Associated with the Statistical Testing Paradigm," Contemporary Accounting Research, John Wiley & Sons, vol. 11(1), pages 33-57, June.
    29. Lindsay, R. Murray, 1995. "Reconsidering the status of tests of significance: An alternative criterion of adequacy," Accounting, Organizations and Society, Elsevier, vol. 20(1), pages 35-53, January.
    30. Robert Bloomfield & Kristina Rennekamp & Blake Steenhoven, 2018. "No System Is Perfect: Understanding How Registration‐Based Editorial Processes Affect Reproducibility and Investment in Research Quality," Journal of Accounting Research, Wiley Blackwell, vol. 56(2), pages 313-362, May.
    31. Basu, Sudipta & Vitanza, Justin & Wang, Wei, 2020. "Asymmetric loan loss provision models," Journal of Accounting and Economics, Elsevier, vol. 70(2).
    32. Thomas R. Dyckman & Stephen A. Zeff, 2015. "Accounting Research: Past, Present, and Future," Abacus, Accounting Foundation, University of Sydney, vol. 51(4), pages 511-524, December.
    33. Johnstone, David, 2018. "Accounting Theory as a Bayesian Discipline," Foundations and Trends(R) in Accounting, now publishers, vol. 13(1-2), pages 1-266, December.
    34. Colin F. Camerer & Anna Dreber & Felix Holzmeister & Teck-Hua Ho & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Gideon Nave & Brian A. Nosek & Thomas Pfeiffer & Adam Altmejd & Nick Buttrick , 2018. "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015," Nature Human Behaviour, Nature, vol. 2(9), pages 637-644, September.
    35. C. Glenn Begley & Lee M. Ellis, 2012. "Raise standards for preclinical cancer research," Nature, Nature, vol. 483(7391), pages 531-533, March.
    36. Paul F. Williams & Sue P. Ravenscroft, 2015. "Rethinking Decision Usefulness," Contemporary Accounting Research, John Wiley & Sons, vol. 32(2), pages 763-788, June.
    37. James A. Ohlson, 2015. "Accounting Research and Common Sense," Abacus, Accounting Foundation, University of Sydney, vol. 51(4), pages 525-535, December.
    38. Jae H. Kim & Kamran Ahmed & Philip Inyeob Ji, 2018. "Significance Testing in Accounting Research: A Critical Evaluation Based on Evidence," Abacus, Accounting Foundation, University of Sydney, vol. 54(4), pages 524-546, December.
    39. Stephen A. Zeff, 2019. "A Personal View of the Evolution of the Accounting Professoriate," Accounting Perspectives, John Wiley & Sons, vol. 18(3), pages 159-185, September.
    40. Thomas R. Dyckman, 2016. "Significance Testing: We Can Do Better," Abacus, Accounting Foundation, University of Sydney, vol. 52(2), pages 319-342, June.
    41. Basu, Sudipta & Markov, Stanimir, 2004. "Loss function assumptions in rational expectations tests on financial analysts' earnings forecasts," Journal of Accounting and Economics, Elsevier, vol. 38(1), pages 171-203, December.
    42. C. Glenn Begley, 2013. "Six red flags for suspect work," Nature, Nature, vol. 497(7450), pages 433-434, May.
    43. David Johnstone, 2016. "The Effect of Information on Uncertainty and the Cost of Capital," Contemporary Accounting Research, John Wiley & Sons, vol. 33(2), pages 752-774, June.
    44. Moizer, Peter, 2009. "Publishing in accounting journals: A fair game?," Accounting, Organizations and Society, Elsevier, vol. 34(2), pages 285-304, February.
    45. Jennifer Francis & Ryan Lafond & Per Olsson & Katherine Schipper, 2007. "Information Uncertainty and Post‐Earnings‐Announcement‐Drift," Journal of Business Finance & Accounting, Wiley Blackwell, vol. 34(3‐4), pages 403-433, April.
    46. Luzi Hail & Mark Lang & Christian Leuz, 2020. "Reproducibility in Accounting Research: Views of the Research Community," Journal of Accounting Research, Wiley Blackwell, vol. 58(2), pages 519-543, May.
    47. Hunter, John E, 2001. "The Desperate Need for Replications," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 28(1), pages 149-158, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Thomas R. Dyckman & Stephen A. Zeff, 2019. "Important Issues in Statistical Testing and Recommended Improvements in Accounting Research," Econometrics, MDPI, vol. 7(2), pages 1-11, May.
    2. Jae H. Kim & Kamran Ahmed & Philip Inyeob Ji, 2018. "Significance Testing in Accounting Research: A Critical Evaluation Based on Evidence," Abacus, Accounting Foundation, University of Sydney, vol. 54(4), pages 524-546, December.
    3. Luzi Hail & Mark Lang & Christian Leuz, 2020. "Reproducibility in Accounting Research: Views of the Research Community," Journal of Accounting Research, Wiley Blackwell, vol. 58(2), pages 519-543, May.
    4. Stewart Jones & Nurul Alam, 2019. "A machine learning analysis of citation impact among selected Pacific Basin journals," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 59(4), pages 2509-2552, December.
    5. James A. Ohlson, 2022. "Researchers’ data analysis choices: an excess of false positives?," Review of Accounting Studies, Springer, vol. 27(2), pages 649-667, June.
    6. Chen, Xiaomeng Charlene & Jones, Stewart & Hasan, Mostafa Monzur & Zhao, Ruoyun & Alam, Nurul, 2023. "Does strategic deviation influence firms’ use of supplier finance?," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 85(C).
    7. Jones, Stewart & Wang, Tim, 2019. "Predicting private company failure: A multi-class analysis," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 61(C), pages 161-188.
    8. Rolf Uwe Fülbier & Thorsten Sellhorn, 2023. "Understanding and improving the language of business: How accounting and corporate reporting research can better serve business and society," Journal of Business Economics, Springer, vol. 93(6), pages 1089-1124, August.
    9. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    10. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    11. Hensel, Przemysław G., 2021. "Reproducibility and replicability crisis: How management compares to psychology and economics – A systematic review of literature," European Management Journal, Elsevier, vol. 39(5), pages 577-594.
    12. Jae H. Kim & In Choi, 2021. "Choosing the Level of Significance: A Decision‐theoretic Approach," Abacus, Accounting Foundation, University of Sydney, vol. 57(1), pages 27-71, March.
    13. Kellner, Ralf & Rösch, Daniel, 2021. "A Bayesian Re-Interpretation of “significant” empirical financial research," Finance Research Letters, Elsevier, vol. 38(C).
    14. Jae H. Kim, 2022. "Moving to a world beyond p-value," Review of Managerial Science, Springer, vol. 16(8), pages 2467-2493, November.
    15. Shiva Rajgopal, 2021. "Integrating Practice into Accounting Research," Management Science, INFORMS, vol. 67(9), pages 5430-5454, September.
    16. Zhang, Sijia & Gregoriou, Andros, 2020. "Post earnings announcement drift, liquidity and zero leverage firms: Evidence from the UK stock market," Journal of Business Research, Elsevier, vol. 116(C), pages 13-26.
    17. Josef Fink, 2020. "A Review of the Post-Earnings-Announcement Drift," Working Paper Series, Social and Economic Sciences 2020-04, Faculty of Social and Economic Sciences, Karl-Franzens-University Graz.
    18. Michaelides, Michael, 2021. "Large sample size bias in empirical finance," Finance Research Letters, Elsevier, vol. 41(C).
    19. Wilkinson, Brett R. & Durden, Chris H., 2015. "Inducing structural change in academic accounting research," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 26(C), pages 23-36.
    20. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:crpeac:v:89:y:2022:i:c:s1045235421000150. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/critical-perspectives-on-accounting/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.