IDEAS home Printed from https://ideas.repec.org/a/bla/abacus/v57y2021i1p27-71.html
   My bibliography  Save this article

Choosing the Level of Significance: A Decision‐theoretic Approach

Author

Listed:
  • Jae H. Kim
  • In Choi

Abstract

In many areas of science, including business disciplines, statistical decisions are often made almost exclusively at a conventional level of significance. Serious concerns have been raised that this contributes to a range of poor practices such as p‐hacking and data‐mining that undermine research credibility. In this paper, we present a decision‐theoretic approach to choosing the optimal level of significance, with a consideration of the key factors of hypothesis testing, including sample size, prior belief, and losses from Type I and II errors. We present the method in the context of testing for linear restrictions in the linear regression model. From the empirical applications in accounting, economics, and finance, we find that the decisions made at the optimal significance levels are more sensible and unambiguous than those at a conventional level, providing inferential outcomes consistent with estimation results, descriptive analysis, and economic reasoning. Computational resources are provided with two R packages.

Suggested Citation

  • Jae H. Kim & In Choi, 2021. "Choosing the Level of Significance: A Decision‐theoretic Approach," Abacus, Accounting Foundation, University of Sydney, vol. 57(1), pages 27-71, March.
    • Handle: RePEc:bla:abacus:v:57:y:2021:i:1:p:27-71
    DOI: 10.1111/abac.12172
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/abac.12172
    Download Restriction: no
    File URL: https://libkey.io/10.1111/abac.12172?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Davidson, Russell & Flachaire, Emmanuel, 2008. "The wild bootstrap, tamed at last," Journal of Econometrics, Elsevier, vol. 146(1), pages 162-169, September.
    2. Shanken, Jay, 1987. "A Bayesian approach to testing portfolio efficiency," Journal of Financial Economics, Elsevier, vol. 19(2), pages 195-215, December.
    3. Startz, Richard, 2014. "Choosing the More Likely Hypothesis," Foundations and Trends(R) in Econometrics, now publishers, vol. 7(2), pages 119-189, November.
    4. Denzil G. Fiebig, 2017. "Big Data: Will It Improve Patient-Centered Care?," The Patient: Patient-Centered Outcomes Research, Springer;International Academy of Health Preference Research, vol. 10(2), pages 133-139, April.
    5. Keuzenkamp, Hugo A. & Magnus, Jan R., 1995. "On tests and significance in econometrics," Journal of Econometrics, Elsevier, vol. 67(1), pages 5-24, May.
    6. Kim, Jae H. & Ji, Philip Inyeob, 2015. "Significance testing in empirical finance: A critical review and assessment," Journal of Empirical Finance, Elsevier, vol. 34(C), pages 1-14.
    7. Fama, Eugene F. & French, Kenneth R., 1993. "Common risk factors in the returns on stocks and bonds," Journal of Financial Economics, Elsevier, vol. 33(1), pages 3-56, February.
    8. Thomas R. Dyckman, 2016. "Significance Testing: We Can Do Better," Abacus, Accounting Foundation, University of Sydney, vol. 52(2), pages 319-342, June.
    9. Tom Engsted, 2009. "Statistical vs. Economic Significance in Economics and Econometrics: Further comments on McCloskey & Ziliak," CREATES Research Papers 2009-17, Department of Economics and Business Economics, Aarhus University.
    10. Soyer, Emre & Hogarth, Robin M., 2012. "The illusion of predictability: How regression statistics mislead experts," International Journal of Forecasting, Elsevier, vol. 28(3), pages 695-711.
    11. Imad A. Moosa, 2017. "Econometrics as a Con Art," Books, Edward Elgar Publishing, number 17257.
    12. Tom Engsted, 2009. "Statistical vs. economic significance in economics and econometrics: further comments on McCloskey and Ziliak," Journal of Economic Methodology, Taylor & Francis Journals, vol. 16(4), pages 393-408.
    13. Das, C., 1994. "Decision making by classical test procedures using an optimal level of significance," European Journal of Operational Research, Elsevier, vol. 73(1), pages 76-84, February.
    14. Fomby, Thomas B. & Guilkey, David K., 1978. "On choosing the optimal level of significance for the Durbin-Watson test and the Bayesian alternative," Journal of Econometrics, Elsevier, vol. 8(2), pages 203-213, October.
    15. Dale J. Poirier, 1995. "Intermediate Statistics and Econometrics: A Comparative Approach," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262161494, December.
    16. Campbell R. Harvey, 2017. "Presidential Address: The Scientific Outlook in Financial Economics," Journal of Finance, American Finance Association, vol. 72(4), pages 1399-1440, August.
    17. David J. Hand, 2016. "Editorial: ‘Big data’ and data sharing," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(3), pages 629-631, June.
    18. Pérez, María-Eglée & Pericchi, Luis Raúl, 2014. "Changing statistical significance with the amount of information: The adaptive α significance level," Statistics & Probability Letters, Elsevier, vol. 85(C), pages 20-24.
    19. Klein, Roger W & Brown, Stephen J, 1984. "Model Selection When There Is "Minimal" Prior Information," Econometrica, Econometric Society, vol. 52(5), pages 1291-1312, September.
    20. Jae H. Kim & Kamran Ahmed & Philip Inyeob Ji, 2018. "Significance Testing in Accounting Research: A Critical Evaluation Based on Evidence," Abacus, Accounting Foundation, University of Sydney, vol. 54(4), pages 524-546, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Run-Ping Che & Mei-Chun Cheung, 2022. "Community-Dwelling Older Adults’ Intended Use of Different Types of Long-Term Care in China and Its Associated Factors Based on the Andersen Behavioral Model," IJERPH, MDPI, vol. 19(18), pages 1-16, September.
    2. Jae H. Kim, 2022. "Moving to a world beyond p-value," Review of Managerial Science, Springer, vol. 16(8), pages 2467-2493, November.
    3. Mahiah Said & Ammar Azfar Azli & Shahnaz Shafiza Sabri, 2023. "Investigating the Factors Influencing Students’ Acceptance of Esports as a Career Choice," Information Management and Business Review, AMH International, vol. 15(2), pages 109-115.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jae H. Kim & Kamran Ahmed & Philip Inyeob Ji, 2018. "Significance Testing in Accounting Research: A Critical Evaluation Based on Evidence," Abacus, Accounting Foundation, University of Sydney, vol. 54(4), pages 524-546, December.
    2. Jae H. Kim & Andrew P. Robinson, 2019. "Interval-Based Hypothesis Testing and Its Applications to Economics and Finance," Econometrics, MDPI, vol. 7(2), pages 1-22, May.
    3. Kim, Jae & Choi, In, 2015. "Unit Roots in Economic and Financial Time Series: A Re-Evaluation based on Enlightened Judgement," MPRA Paper 68411, University Library of Munich, Germany.
    4. Jae H. Kim & In Choi, 2017. "Unit Roots in Economic and Financial Time Series: A Re-Evaluation at the Decision-Based Significance Levels," Econometrics, MDPI, vol. 5(3), pages 1-23, September.
    5. Kim, Jae H. & Ji, Philip Inyeob, 2015. "Significance testing in empirical finance: A critical review and assessment," Journal of Empirical Finance, Elsevier, vol. 34(C), pages 1-14.
    6. Kim, Jae, 2015. "How to Choose the Level of Significance: A Pedagogical Note," MPRA Paper 66373, University Library of Munich, Germany.
    7. Jae H. Kim, 2022. "Moving to a world beyond p-value," Review of Managerial Science, Springer, vol. 16(8), pages 2467-2493, November.
    8. Muhammad Ishaq Bhatti & Jae H. Kim, 2020. "Towards a New Paradigm for Statistical Evidence in the Use of p -Value," Econometrics, MDPI, vol. 9(1), pages 1-3, December.
    9. Johnstone, David, 2022. "Accounting research and the significance test crisis," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 89(C).
    10. Kellner, Ralf & Rösch, Daniel, 2021. "A Bayesian Re-Interpretation of “significant” empirical financial research," Finance Research Letters, Elsevier, vol. 38(C).
    11. Thomas Mayer, 2012. "Ziliak and McCloskey's Criticisms of Significance Tests: An Assessment," Econ Journal Watch, Econ Journal Watch, vol. 9(3), pages 256-297, September.
    12. Kim, Jae H., 2017. "Stock returns and investors' mood: Good day sunshine or spurious correlation?," International Review of Financial Analysis, Elsevier, vol. 52(C), pages 94-103.
    13. Stewart Jones & Nurul Alam, 2019. "A machine learning analysis of citation impact among selected Pacific Basin journals," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 59(4), pages 2509-2552, December.
    14. Michaelides, Michael, 2021. "Large sample size bias in empirical finance," Finance Research Letters, Elsevier, vol. 41(C).
    15. Chinco, Alex & Neuhierl, Andreas & Weber, Michael, 2021. "Estimating the anomaly base rate," Journal of Financial Economics, Elsevier, vol. 140(1), pages 101-126.
    16. Fletcher, Jonathan, 2018. "Bayesian tests of global factor models," Journal of Empirical Finance, Elsevier, vol. 48(C), pages 279-289.
    17. Pastor, Lubos & Stambaugh, Robert F., 2000. "Comparing asset pricing models: an investment perspective," Journal of Financial Economics, Elsevier, vol. 56(3), pages 335-381, June.
    18. Fieberg, Christian & Günther, Steffen & Poddig, Thorsten & Zaremba, Adam, 2024. "Non-standard errors in the cryptocurrency world," International Review of Financial Analysis, Elsevier, vol. 92(C).
    19. James A. Ohlson, 2022. "Researchers’ data analysis choices: an excess of false positives?," Review of Accounting Studies, Springer, vol. 27(2), pages 649-667, June.
    20. Chen, Xiaomeng Charlene & Jones, Stewart & Hasan, Mostafa Monzur & Zhao, Ruoyun & Alam, Nurul, 2023. "Does strategic deviation influence firms’ use of supplier finance?," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 85(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:abacus:v:57:y:2021:i:1:p:27-71. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.blackwellpublishing.com/journal.asp?ref=0001-3072 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.