IDEAS home Printed from https://ideas.repec.org/a/zbw/jcreco/267164.html

A Proposed Hybrid Effect Size Plus p-Value Criterion. A Comment on Goodman et al. (The American Statistician, 2019)

Author

Listed:
  • Pütz, Peter
  • Kramer-Sunderbrink, Arne
  • Dreher, Robin Tim
  • Hoffmann, Leona
  • Werner, Robin

Abstract

In a recent simulation study, Goodman et al. (2019) compare several methods with regard to their type I and type II error rates when considering a thick null hypothesis that includes all values that are practically equivalent to the point null hypothesis. They propose a hybrid decision criterion only declaring a result "significant" if both a small p-value and a sufficiently large effect size are obtained. We successfully verify the results using our own software code in R and discuss an additional decision method that is tailored to maintain a pre-defined false positive rate. We confirm that the hybrid decision criterion has comparably low error rates in checkable settings but point out that the false discovery rate cannot be easily controlled by the researcher. Our analyses are readily accessible and customizable on github.com/drehero/goodman-replication.

Suggested Citation

  • Pütz, Peter & Kramer-Sunderbrink, Arne & Dreher, Robin Tim & Hoffmann, Leona & Werner, Robin, 2022. "A Proposed Hybrid Effect Size Plus p-Value Criterion. A Comment on Goodman et al. (The American Statistician, 2019)," Journal of Comments and Replications in Economics (JCRE), ZBW - Leibniz Information Centre for Economics, vol. 1, pages 1-15.
    • Handle: RePEc:zbw:jcreco:267164
    DOI: 10.18718/81781.26
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/267164/1/JCRE-2022-4.pdf
    Download Restriction: no
    File URL: https://libkey.io/10.18718/81781.26?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Sander Greenland, 2019. "Valid P-Values Behave Exactly as They Should: Some Misleading Criticisms of P-Values and Their Resolution With S-Values," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 106-114, March.
    2. Ronald L. Wasserstein & Allen L. Schirm & Nicole A. Lazar, 2019. "Moving to a World Beyond “p," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 1-19, March.
    3. Rebecca A. Betensky, 2019. "The p-Value Requires Context, Not a Threshold," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 115-117, March.
    4. Daniel J. Benjamin & James O. Berger & Magnus Johannesson & Brian A. Nosek & E.-J. Wagenmakers & Richard Berk & Kenneth A. Bollen & Björn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Chr, 2018. "Redefine statistical significance," Nature Human Behaviour, Nature, vol. 2(1), pages 6-10, January.
      • Daniel Benjamin & James Berger & Magnus Johannesson & Brian Nosek & E. Wagenmakers & Richard Berk & Kenneth Bollen & Bjorn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Christopher Chambe, 2017. "Redefine Statistical Significance," Artefactual Field Experiments 00612, The Field Experiments Website.
    5. Jeffrey D. Blume & Robert A. Greevy & Valerie F. Welty & Jeffrey R. Smith & William D. Dupont, 2019. "An Introduction to Second-Generation p-Values," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 157-167, March.
    6. William M. Goodman & Susan E. Spruill & Eugene Komaroff, 2019. "A Proposed Hybrid Effect Size Plus p-Value Criterion: Empirical Evidence Supporting its Use," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 168-185, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alain Bultez & Jean-Luc Herrmann, 2025. "Value added to marketing research diagnoses by add-ons to $${\varvec{p}}$$ p -values," Journal of Marketing Analytics, Palgrave Macmillan, vol. 13(2), pages 445-466, June.
    2. Gunter, Ulrich & Önder, Irem & Smeral, Egon, 2019. "Scientific value of econometric tourism demand studies," Annals of Tourism Research, Elsevier, vol. 78(C), pages 1-1.
    3. Grüner Sven, 2020. "Sample Size Calculation in Economic Experiments," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 240(6), pages 791-823, December.
    4. Stephane Hess & Andrew Daly & Michiel Bliemer & Angelo Guevara & Ricardo Daziano & Thijs Dekker, 2025. "Statistical significance in choice modelling: computation, usage and reporting," Papers 2506.05996, arXiv.org, revised May 2026.
    5. Kelter, Riko, 2022. "Power analysis and type I and type II error rates of Bayesian nonparametric two-sample tests for location-shifts based on the Bayes factor under Cauchy priors," Computational Statistics & Data Analysis, Elsevier, vol. 165(C).
    6. Markku Maula & Wouter Stam, 2020. "Enhancing Rigor in Quantitative Entrepreneurship Research," Entrepreneurship Theory and Practice, , vol. 44(6), pages 1059-1090, November.
    7. Riko Kelter, 2024. "The Case of the Jeffreys-Lindley-paradox as a Bayes-frequentist Compromise: A Perspective Based on the Rao-Lovric-Theorem," Sankhya A: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 86(1), pages 337-363, February.
    8. Eleni Verykouki & Christos T. Nakas, 2023. "Adaptations on the Use of p -Values for Statistical Inference: An Interpretation of Messages from Recent Public Discussions," Stats, MDPI, vol. 6(2), pages 1-13, April.
    9. Glenn Shafer, 2021. "Testing by betting: A strategy for statistical and scientific communication," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(2), pages 407-431, April.
    10. Rigdon, Edward E., 2023. "How improper dichotomization and the misrepresentation of uncertainty undermine social science research," Journal of Business Research, Elsevier, vol. 165(C).
    11. David R. Bickel, 2024. "The p-value interpreted as the posterior probability of explaining the data: Applications to multiple testing and to restricted parameter spaces," Sankhya A: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 86(1), pages 464-493, February.
    12. Eric-Jan Wagenmakers & Alexandra Sarafoglou & Sil Aarts & Casper Albers & Johannes Algermissen & Štěpán Bahník & Noah Dongen & Rink Hoekstra & David Moreau & Don Ravenzwaaij & Aljaž Sluga & Franziska , 2021. "Seven steps toward more transparency in statistical practice," Nature Human Behaviour, Nature, vol. 5(11), pages 1473-1480, November.
    13. Jae H. Kim, 2022. "Moving to a world beyond p-value," Review of Managerial Science, Springer, vol. 16(8), pages 2467-2493, November.
    14. Erik W. van Zwet & Eric A. Cator, 2021. "The significance filter, the winner's curse and the need to shrink," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 75(4), pages 437-452, November.
    15. Fukś Maksymilian & Wiejaczka Łukasz, 2025. "Climatic Determinants of Changes in the Ice Regime of Carpathian Rivers," Quaestiones Geographicae, Paradigm, vol. 44(1), pages 131-143.
    16. Mayo, Deborah, 2024. "Error statistics, Bayes-factor Tests and the Fallacy of Non-exhaustive Alternatives," OSF Preprints tmgqd, Center for Open Science.
    17. Dreber, Anna & Heikensten, Emma & Säve-Söderbergh, Jenny, 2022. "Why do women ask for less?," Labour Economics, Elsevier, vol. 78(C).
    18. Ruijun Hou & Samuel Baker & Stephanie von Hinke & Hans H. Sievertsen & Emil Sorensen & Nicolai Vitt, 2025. "Long-term Health and Human Capital Effects of EarlyLife Economic Conditions," Bristol Economics Discussion Papers 25/814, School of Economics, University of Bristol, UK.
    19. Ardila Brenøe, Anne & Eyibak, Zeynep & Heursen, Lea & Ranehill, Eva & Weber, Roberto A., 2024. "Gender Identity and Economic Decision Making," Working Papers 2024:6, Lund University, Department of Economics.
    20. Buchanan, Erin Michelle & Scofield, John E., 2017. "Bulletproof Bias? Considering the Type of Data in Common Proportion of Variance Effect Sizes," OSF Preprints cs4vy, Center for Open Science.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;

    JEL classification:

    • C10 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - General
    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:jcreco:267164. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/zbwkide.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.