IDEAS home Printed from https://ideas.repec.org/a/bla/abacus/v52y2016i2p319-342.html
   My bibliography  Save this article

Significance Testing: We Can Do Better

Author

Listed:
  • Thomas R. Dyckman

Abstract

type="main"> This paper advocates abandoning null hypothesis statistical tests (NHST) in favour of reporting confidence intervals. The case against NHST, which has been made repeatedly in multiple disciplines and is growing in awareness and acceptance, is introduced and discussed. Accounting as an empirical research discipline appears to be the last of the research communities to face up to the inherent problems of significance test use and abuse. The paper encourages adoption of a meta-analysis approach which allows for the inclusion of replication studies in the assessment of evidence. This approach requires abandoning the typical NHST process and its reliance on p-values. However, given that NHST has deep roots and wide ‘social acceptance’ in the empirical testing community, modifications to NHST are suggested so as to partly counter the weakness of this statistical testing method.

Suggested Citation

  • Thomas R. Dyckman, 2016. "Significance Testing: We Can Do Better," Abacus, Accounting Foundation, University of Sydney, vol. 52(2), pages 319-342, June.
    • Handle: RePEc:bla:abacus:v:52:y:2016:i:2:p:319-342
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1111/abac.12078
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Paldam, Martin, 2015. "Meta-analysis in a nutshell: Techniques and general findings," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 9, pages 1-14.
    2. Fidler, Fiona & Geoff, Cumming & Mark, Burgman & Neil, Thomason, 2004. "Statistical reform in medicine, psychology and ecology," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(5), pages 615-630, November.
    3. Nalan Basturk & Cem Cakmakli & S. Pinar Ceyhan & Herman K. van Dijk, 2014. "On the Rise of Bayesian Econometrics after Cowles Foundation Monographs 10, 14," Tinbergen Institute Discussion Papers 14-085/III, Tinbergen Institute, revised 04 Sep 2014.
    4. Ziliak, Stephen T. & McCloskey, Deirdre N., 2004. "Size matters: the standard error of regressions in the American Economic Review," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(5), pages 527-546, November.
    5. David Cox & Deborah Mayo, 2011. "Statistical Scientist Meets a Philosopher of Science: A Conversation," Rationality, Markets and Morals, Frankfurt School Verlag, Frankfurt School of Finance & Management, vol. 2(45), October.
    6. R. Murray Lindsay, 1994. "Publication System Biases Associated with the Statistical Testing Paradigm," Contemporary Accounting Research, John Wiley & Sons, vol. 11(1), pages 33-57, June.
    7. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
    8. Lindsay, R. Murray, 1995. "Reconsidering the status of tests of significance: An alternative criterion of adequacy," Accounting, Organizations and Society, Elsevier, vol. 20(1), pages 35-53, January.
    9. D. J. Johnstone, 1997. "Comparative Classical and Bayesian Interpretations of Statistical Compliance Tests in Auditing," Accounting and Business Research, Taylor & Francis Journals, vol. 28(1), pages 53-82, July.
    10. repec:dau:papers:123456789/1908 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yi Jiang & Stewart Jones, 2018. "Corporate distress prediction in China: a machine learning approach," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 58(4), pages 1063-1109, December.
    2. Erik Johannesson & James A. Ohlson & Sophia Weihuan Zhai, 2024. "The explanatory power of explanatory variables," Review of Accounting Studies, Springer, vol. 29(4), pages 3053-3083, December.
    3. Chen, Xiaomeng Charlene & Jones, Stewart & Hasan, Mostafa Monzur & Zhao, Ruoyun & Alam, Nurul, 2023. "Does strategic deviation influence firms’ use of supplier finance?," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 85(C).
    4. Jae H. Kim & In Choi, 2021. "Choosing the Level of Significance: A Decision‐theoretic Approach," Abacus, Accounting Foundation, University of Sydney, vol. 57(1), pages 27-71, March.
    5. Jae H. Kim & Kamran Ahmed & Philip Inyeob Ji, 2018. "Significance Testing in Accounting Research: A Critical Evaluation Based on Evidence," Abacus, Accounting Foundation, University of Sydney, vol. 54(4), pages 524-546, December.
    6. James A. Ohlson, 2022. "Researchers’ data analysis choices: an excess of false positives?," Review of Accounting Studies, Springer, vol. 27(2), pages 649-667, June.
    7. Jones, Stewart & Wang, Tim, 2019. "Predicting private company failure: A multi-class analysis," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 61(C), pages 161-188.
    8. Johnstone, David, 2022. "Accounting research and the significance test crisis," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 89(C).
    9. Thomas R. Dyckman & Stephen A. Zeff, 2019. "Important Issues in Statistical Testing and Recommended Improvements in Accounting Research," Econometrics, MDPI, vol. 7(2), pages 1-11, May.
    10. Mostafa Monzur Hasan & Grantley Taylor & Grant Richardson, 2022. "Brand Capital and Stock Price Crash Risk," Management Science, INFORMS, vol. 68(10), pages 7221-7247, October.
    11. Stewart Jones & Nurul Alam, 2019. "A machine learning analysis of citation impact among selected Pacific Basin journals," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 59(4), pages 2509-2552, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Thomas R. Dyckman & Stephen A. Zeff, 2019. "Important Issues in Statistical Testing and Recommended Improvements in Accounting Research," Econometrics, MDPI, vol. 7(2), pages 1-11, May.
    2. Altman, Morris, 2004. "Statistical significance, path dependency, and the culture of journal publication," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(5), pages 651-663, November.
    3. Marc Orlitzky, 2011. "Institutionalized dualism: statistical significance testing as myth and ceremony," Metrika: International Journal for Theoretical and Applied Statistics, Springer, vol. 22(1), pages 47-77, September.
    4. Gunter, Ulrich & Önder, Irem & Smeral, Egon, 2019. "Scientific value of econometric tourism demand studies," Annals of Tourism Research, Elsevier, vol. 78(C), pages 1-1.
    5. Ho, Carol-Anne & Williams, S. Mitchell, 2003. "Reply to "International comparative analysis of the association between board structure and the efficiency of value added by a firm from its physical capital and intellectual capital resources: A," The International Journal of Accounting, Elsevier, vol. 38(4), pages 499-502.
    6. Paldam, Martin, 2018. "A model of the representative economist, as researcher and policy advisor," European Journal of Political Economy, Elsevier, vol. 54(C), pages 5-15.
    7. Beverley R. Lord & Yvonne P. Shanahan & Benjamin M. Nolan, 2007. "The Use and Perceived Merit of Customer Accounting in New Zealand," Accounting Research Journal, Emerald Group Publishing Limited, vol. 20(1), pages 47-59, July.
    8. Jae H. Kim & Kamran Ahmed & Philip Inyeob Ji, 2018. "Significance Testing in Accounting Research: A Critical Evaluation Based on Evidence," Abacus, Accounting Foundation, University of Sydney, vol. 54(4), pages 524-546, December.
    9. John D. Levendis, 2018. "Time Series Econometrics," Springer Texts in Business and Economics, Springer, number 978-3-319-98282-3, March.
    10. Smith Bamber, Linda & Christensen, Theodore E. & Gaver, Kenneth M., 2000. "Do we really 'know' what we think we know? A case study of seminal research and its subsequent overgeneralization," Accounting, Organizations and Society, Elsevier, vol. 25(2), pages 103-129, February.
    11. Andrea Ongaro & Sonia Migliorati & Roberto Ascari & Enrico Ripamonti, 2024. "Testing practical relevance of treatment effects," Statistical Papers, Springer, vol. 65(7), pages 4121-4145, September.
    12. Hubbard, Raymond & Lindsay, R. Murray, 2013. "From significant difference to significant sameness: Proposing a paradigm shift in business research," Journal of Business Research, Elsevier, vol. 66(9), pages 1377-1388.
    13. Johnstone, David, 2022. "Accounting research and the significance test crisis," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 89(C).
    14. Klaus Derfuss, 2015. "Relating Context Variables to Participative Budgeting and Evaluative Use of Performance Measures: A Meta-analysis," Abacus, Accounting Foundation, University of Sydney, vol. 51(2), pages 238-278, June.
    15. Campbell R. Harvey & Yan Liu, 2020. "False (and Missed) Discoveries in Financial Economics," Journal of Finance, American Finance Association, vol. 75(5), pages 2503-2553, October.
    16. Jing Xing, 2011. "Does tax structure affect economic growth? Empirical evidence from OECD countries," Working Papers 1120, Oxford University Centre for Business Taxation.
    17. Rachel G. Childers, 2011. "Being One'S Own Boss: How Does Risk Fit In?," The American Economist, Sage Publications, vol. 56(1), pages 48-58, May.
    18. Marc-Olivier Bessette & Mariame Dioubate & Myriane Hébert & Miriam Elsie Kuimi Tchana & Laura Morissette & Jean-Charles Toupin & Raoul Yaro & Maurice Doyon, 2020. "La présence de biais cognitifs en analyse économique : une étude de cas," CIRANO Working Papers 2020s-12, CIRANO.
    19. Christoph Engel, 2006. "The Difficult Reception of Rigorous Descriptive Social Science in the Law," Discussion Paper Series of the Max Planck Institute for Research on Collective Goods 2006_1, Max Planck Institute for Research on Collective Goods.
    20. Durlauf, Steven N., 2001. "Manifesto for a growth econometrics," Journal of Econometrics, Elsevier, vol. 100(1), pages 65-69, January.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:abacus:v:52:y:2016:i:2:p:319-342. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.blackwellpublishing.com/journal.asp?ref=0001-3072 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.