IDEAS home Printed from https://ideas.repec.org/a/bla/abacus/v52y2016i2p319-342.html
   My bibliography  Save this article

Significance Testing: We Can Do Better

Author

Listed:
  • Thomas R. Dyckman

Abstract

type="main"> This paper advocates abandoning null hypothesis statistical tests (NHST) in favour of reporting confidence intervals. The case against NHST, which has been made repeatedly in multiple disciplines and is growing in awareness and acceptance, is introduced and discussed. Accounting as an empirical research discipline appears to be the last of the research communities to face up to the inherent problems of significance test use and abuse. The paper encourages adoption of a meta-analysis approach which allows for the inclusion of replication studies in the assessment of evidence. This approach requires abandoning the typical NHST process and its reliance on p-values. However, given that NHST has deep roots and wide ‘social acceptance’ in the empirical testing community, modifications to NHST are suggested so as to partly counter the weakness of this statistical testing method.

Suggested Citation

  • Thomas R. Dyckman, 2016. "Significance Testing: We Can Do Better," Abacus, Accounting Foundation, University of Sydney, vol. 52(2), pages 319-342, June.
    • Handle: RePEc:bla:abacus:v:52:y:2016:i:2:p:319-342
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1111/abac.12078
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Stephen T. Ziliak & Deirdre N. McCloskey, 2004. "Size Matters: The Standard Error of Regressions in the American Economic Review," Econ Journal Watch, Econ Journal Watch, vol. 1(2), pages 331-358, August.
    2. Paldam, Martin, 2015. "Meta-analysis in a nutshell: Techniques and general findings," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 9, pages 1-14.
    3. Fidler, Fiona & Geoff, Cumming & Mark, Burgman & Neil, Thomason, 2004. "Statistical reform in medicine, psychology and ecology," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(5), pages 615-630, November.
    4. R. Murray Lindsay, 1994. "Publication System Biases Associated with the Statistical Testing Paradigm," Contemporary Accounting Research, John Wiley & Sons, vol. 11(1), pages 33-57, June.
    5. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
    6. Lindsay, R. Murray, 1995. "Reconsidering the status of tests of significance: An alternative criterion of adequacy," Accounting, Organizations and Society, Elsevier, vol. 20(1), pages 35-53, January.
    7. repec:dau:papers:123456789/1908 is not listed on IDEAS
    8. David Cox & Deborah Mayo, 2011. "Statistical Scientist Meets a Philosopher of Science: A Conversation," Rationality, Markets and Morals, Frankfurt School Verlag, Frankfurt School of Finance & Management, vol. 2(45), October.
    9. Nalan Basturk & Cem Cakmakli & S. Pinar Ceyhan & Herman K. van Dijk, 2014. "On the Rise of Bayesian Econometrics after Cowles Foundation Monographs 10, 14," Tinbergen Institute Discussion Papers 14-085/III, Tinbergen Institute, revised 04 Sep 2014.
    10. D. J. Johnstone, 1997. "Comparative Classical and Bayesian Interpretations of Statistical Compliance Tests in Auditing," Accounting and Business Research, Taylor & Francis Journals, vol. 28(1), pages 53-82, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yi Jiang & Stewart Jones, 2018. "Corporate distress prediction in China: a machine learning approach," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 58(4), pages 1063-1109, December.
    2. Jones, Stewart & Wang, Tim, 2019. "Predicting private company failure: A multi-class analysis," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 61(C), pages 161-188.
    3. Johnstone, David, 2022. "Accounting research and the significance test crisis," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 89(C).
    4. Thomas R. Dyckman & Stephen A. Zeff, 2019. "Important Issues in Statistical Testing and Recommended Improvements in Accounting Research," Econometrics, MDPI, vol. 7(2), pages 1-11, May.
    5. Mostafa Monzur Hasan & Grantley Taylor & Grant Richardson, 2022. "Brand Capital and Stock Price Crash Risk," Management Science, INFORMS, vol. 68(10), pages 7221-7247, October.
    6. Chen, Xiaomeng Charlene & Jones, Stewart & Hasan, Mostafa Monzur & Zhao, Ruoyun & Alam, Nurul, 2023. "Does strategic deviation influence firms’ use of supplier finance?," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 85(C).
    7. Jae H. Kim & In Choi, 2021. "Choosing the Level of Significance: A Decision‐theoretic Approach," Abacus, Accounting Foundation, University of Sydney, vol. 57(1), pages 27-71, March.
    8. Jae H. Kim & Kamran Ahmed & Philip Inyeob Ji, 2018. "Significance Testing in Accounting Research: A Critical Evaluation Based on Evidence," Abacus, Accounting Foundation, University of Sydney, vol. 54(4), pages 524-546, December.
    9. James A. Ohlson, 2022. "Researchers’ data analysis choices: an excess of false positives?," Review of Accounting Studies, Springer, vol. 27(2), pages 649-667, June.
    10. Stewart Jones & Nurul Alam, 2019. "A machine learning analysis of citation impact among selected Pacific Basin journals," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 59(4), pages 2509-2552, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Thomas R. Dyckman & Stephen A. Zeff, 2019. "Important Issues in Statistical Testing and Recommended Improvements in Accounting Research," Econometrics, MDPI, vol. 7(2), pages 1-11, May.
    2. Jae H. Kim & Kamran Ahmed & Philip Inyeob Ji, 2018. "Significance Testing in Accounting Research: A Critical Evaluation Based on Evidence," Abacus, Accounting Foundation, University of Sydney, vol. 54(4), pages 524-546, December.
    3. Altman, Morris, 2004. "Statistical significance, path dependency, and the culture of journal publication," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(5), pages 651-663, November.
    4. Marc Orlitzky, 2011. "Institutionalized dualism: statistical significance testing as myth and ceremony," Metrika: International Journal for Theoretical and Applied Statistics, Springer, vol. 22(1), pages 47-77, September.
    5. John D. Levendis, 2018. "Time Series Econometrics," Springer Texts in Business and Economics, Springer, number 978-3-319-98282-3, August.
    6. Gunter, Ulrich & Önder, Irem & Smeral, Egon, 2019. "Scientific value of econometric tourism demand studies," Annals of Tourism Research, Elsevier, vol. 78(C), pages 1-1.
    7. Smith Bamber, Linda & Christensen, Theodore E. & Gaver, Kenneth M., 2000. "Do we really 'know' what we think we know? A case study of seminal research and its subsequent overgeneralization," Accounting, Organizations and Society, Elsevier, vol. 25(2), pages 103-129, February.
    8. Ho, Carol-Anne & Williams, S. Mitchell, 2003. "Reply to "International comparative analysis of the association between board structure and the efficiency of value added by a firm from its physical capital and intellectual capital resources: A," The International Journal of Accounting, Elsevier, vol. 38(4), pages 499-502.
    9. Hubbard, Raymond & Lindsay, R. Murray, 2013. "From significant difference to significant sameness: Proposing a paradigm shift in business research," Journal of Business Research, Elsevier, vol. 66(9), pages 1377-1388.
    10. Johnstone, David, 2022. "Accounting research and the significance test crisis," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 89(C).
    11. Paldam, Martin, 2018. "A model of the representative economist, as researcher and policy advisor," European Journal of Political Economy, Elsevier, vol. 54(C), pages 5-15.
    12. Beverley R. Lord & Yvonne P. Shanahan & Benjamin M. Nolan, 2007. "The Use and Perceived Merit of Customer Accounting in New Zealand," Accounting Research Journal, Emerald Group Publishing Limited, vol. 20(1), pages 47-59, July.
    13. Klaus Derfuss, 2015. "Relating Context Variables to Participative Budgeting and Evaluative Use of Performance Measures: A Meta-analysis," Abacus, Accounting Foundation, University of Sydney, vol. 51(2), pages 238-278, June.
    14. Campbell R. Harvey & Yan Liu, 2020. "False (and Missed) Discoveries in Financial Economics," Journal of Finance, American Finance Association, vol. 75(5), pages 2503-2553, October.
    15. Cho, Seo-young & Vadlamannati, Krishna Chaitanya, 2010. "Compliance for big brothers: An empirical analysis on the impact of the anti-trafficking protocol," University of Göttingen Working Papers in Economics 118, University of Goettingen, Department of Economics.
    16. Ho, Ron Yiu-wah & Strange, Roger & Piesse, Jenifer, 2006. "On the conditional pricing effects of beta, size, and book-to-market equity in the Hong Kong market," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 16(3), pages 199-214, July.
    17. Jing Xing, 2011. "Does tax structure affect economic growth? Empirical evidence from OECD countries," Working Papers 1120, Oxford University Centre for Business Taxation.
    18. Rachel G. Childers, 2011. "Being One'S Own Boss: How Does Risk Fit In?," The American Economist, Sage Publications, vol. 56(1), pages 48-58, May.
    19. Ricardo Barradas & Ines Tomas, 2023. "Household indebtedness in the European Union countries: Going beyond the mainstream interpretation," PSL Quarterly Review, Economia civile, vol. 76(304), pages 21-49.
    20. David Card, 2022. "Design-Based Research in Empirical Microeconomics," American Economic Review, American Economic Association, vol. 112(6), pages 1773-1781, June.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:abacus:v:52:y:2016:i:2:p:319-342. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.blackwellpublishing.com/journal.asp?ref=0001-3072 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.