IDEAS home Printed from https://ideas.repec.org/a/zbw/jcreco/267164.html
   My bibliography  Save this article

A Proposed Hybrid Effect Size Plus p-Value Criterion. A Comment on Goodman et al. (The American Statistician, 2019)

Author

Listed:
  • Pütz, Peter
  • Kramer-Sunderbrink, Arne
  • Dreher, Robin Tim
  • Hoffmann, Leona
  • Werner, Robin

Abstract

In a recent simulation study, Goodman et al. (2019) compare several methods with regard to their type I and type II error rates when considering a thick null hypothesis that includes all values that are practically equivalent to the point null hypothesis. They propose a hybrid decision criterion only declaring a result "significant" if both a small p-value and a sufficiently large effect size are obtained. We successfully verify the results using our own software code in R and discuss an additional decision method that is tailored to maintain a pre-defined false positive rate. We confirm that the hybrid decision criterion has comparably low error rates in checkable settings but point out that the false discovery rate cannot be easily controlled by the researcher. Our analyses are readily accessible and customizable on github.com/drehero/goodman-replication.

Suggested Citation

  • Pütz, Peter & Kramer-Sunderbrink, Arne & Dreher, Robin Tim & Hoffmann, Leona & Werner, Robin, 2022. "A Proposed Hybrid Effect Size Plus p-Value Criterion. A Comment on Goodman et al. (The American Statistician, 2019)," Journal of Comments and Replications in Economics (JCRE), ZBW - Leibniz Information Centre for Economics, vol. 1(2022-4), pages 1-15.
    • Handle: RePEc:zbw:jcreco:267164
    DOI: 10.18718/81781.26
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/267164/1/JCRE-2022-4.pdf
    Download Restriction: no
    File URL: https://libkey.io/10.18718/81781.26?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Sander Greenland, 2019. "Valid P-Values Behave Exactly as They Should: Some Misleading Criticisms of P-Values and Their Resolution With S-Values," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 106-114, March.
    2. William M. Goodman & Susan E. Spruill & Eugene Komaroff, 2019. "A Proposed Hybrid Effect Size Plus p-Value Criterion: Empirical Evidence Supporting its Use," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 168-185, March.
    3. Ronald L. Wasserstein & Allen L. Schirm & Nicole A. Lazar, 2019. "Moving to a World Beyond “p," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 1-19, March.
    4. Rebecca A. Betensky, 2019. "The p-Value Requires Context, Not a Threshold," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 115-117, March.
    5. Jeffrey D. Blume & Robert A. Greevy & Valerie F. Welty & Jeffrey R. Smith & William D. Dupont, 2019. "An Introduction to Second-Generation p-Values," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 157-167, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Markku Maula & Wouter Stam, 2020. "Enhancing Rigor in Quantitative Entrepreneurship Research," Entrepreneurship Theory and Practice, , vol. 44(6), pages 1059-1090, November.
    2. Gunter, Ulrich & Önder, Irem & Smeral, Egon, 2019. "Scientific value of econometric tourism demand studies," Annals of Tourism Research, Elsevier, vol. 78(C), pages 1-1.
    3. Eleni Verykouki & Christos T. Nakas, 2023. "Adaptations on the Use of p -Values for Statistical Inference: An Interpretation of Messages from Recent Public Discussions," Stats, MDPI, vol. 6(2), pages 1-13, April.
    4. Jae H. Kim, 2022. "Moving to a world beyond p-value," Review of Managerial Science, Springer, vol. 16(8), pages 2467-2493, November.
    5. Gruener, Sven, 2019. "An empirical study on Internet-based false news stories: experiences, problem awareness, and responsibilities," SocArXiv xbez9, Center for Open Science.
    6. Maria-Lavinia FLOREA & Anca BORZA, 2019. "Individual Strategies For Achieving Work-Life Balance €“ A Case Study On Romanian Workers," Proceedings of the INTERNATIONAL MANAGEMENT CONFERENCE, Faculty of Management, Academy of Economic Studies, Bucharest, Romania, vol. 13(1), pages 845-855, November.
    7. Thomas R. Dyckman & Stephen A. Zeff, 2019. "Important Issues in Statistical Testing and Recommended Improvements in Accounting Research," Econometrics, MDPI, vol. 7(2), pages 1-11, May.
    8. Grüner Sven, 2020. "Sample Size Calculation in Economic Experiments," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 240(6), pages 791-823, December.
    9. Azeem, Muhammad Masood & Sheridan, Alison & Adapa, Sujana, 2022. "Women to women: Enabling innovation and firm performance in developing countries," Emerging Markets Review, Elsevier, vol. 51(PA).
    10. Michaelides, Michael, 2021. "Large sample size bias in empirical finance," Finance Research Letters, Elsevier, vol. 41(C).
    11. Kelter, Riko, 2022. "Power analysis and type I and type II error rates of Bayesian nonparametric two-sample tests for location-shifts based on the Bayes factor under Cauchy priors," Computational Statistics & Data Analysis, Elsevier, vol. 165(C).
    12. John Cawley & Alex Susskind & Barton Willage, 2020. "The Impact of Information Disclosure on Consumer Behavior: Evidence from a Randomized Field Experiment of Calorie Labels on Restaurant Menus," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(4), pages 1020-1042, September.
    13. Liu, Yu & Maula, Markku, 2021. "Contextual status effects: The performance effects of host-country network status and regulatory institutions in cross-border venture capital," Research Policy, Elsevier, vol. 50(5).
    14. Abbas, Sadia & Adapa, Sujana & Sheridan, Alison & Azeem, Muhammad Masood, 2022. "Informal competition and firm level innovation in South Asia: The moderating role of innovation time off and R&D intensity," Technological Forecasting and Social Change, Elsevier, vol. 181(C).
    15. Michal Lehnert & Mark De Ste Croix & Amr Zaatar & Patrycja Lipinska & Petr Stastny, 2020. "Effect of a Simulated Match on Lower Limb Neuromuscular Performance in Youth Footballers—A Two Year Longitudinal Study," IJERPH, MDPI, vol. 17(22), pages 1-17, November.
    16. Borenstein, Denis & Perlin, Marcelo S. & Imasato, Takeyoshi, 2022. "The Academic Inbreeding Controversy: Analysis and Evidence from Brazil," Journal of Informetrics, Elsevier, vol. 16(2).
    17. Simon Berset & Martin Huber & Mark Schelker, 2023. "The fiscal response to revenue shocks," International Tax and Public Finance, Springer;International Institute of Public Finance, vol. 30(3), pages 814-848, June.
    18. Cheng, Yuanyuan, 2023. "A method of 3R to evaluate the correlation and predictive value of variables," OSF Preprints c79tu, Center for Open Science.
    19. Heckelei, Thomas & Huettel, Silke & Odening, Martin & Rommel, Jens, 2021. "The replicability crisis and the p-value debate – what are the consequences for the agricultural and food economics community?," Discussion Papers 316369, University of Bonn, Institute for Food and Resource Economics.
    20. Jordan Adamson & Lucas Rentschler, 2023. "Criminal justice from a public choice perspective: an introduction to the special issue," Public Choice, Springer, vol. 196(3), pages 223-227, September.

    More about this item

    Keywords

    NHST; Intervall null hypothesis; Minimum effect size plus p-value criterion; Thick t-test; Statistical evidence; Type I error rate; False discovery rate;
    All these keywords.

    JEL classification:

    • C10 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - General
    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:jcreco:267164. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/zbwkide.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.