IDEAS home Printed from https://ideas.repec.org/a/bla/stanee/v77y2023i4p573-591.html

Assessing replicability with the sceptical p$$ p $$‐value: Type‐I error control and sample size planning

Author

Listed:
  • Charlotte Micheloud
  • Fadoua Balabdaoui
  • Leonhard Held

Abstract

We study a statistical framework for replicability based on a recently proposed quantitative measure of replication success, the sceptical p$$ p $$‐value. A recalibration is proposed to obtain exact overall Type‐I error control if the effect is null in both studies and additional bounds on the partial and conditional Type‐I error rate, which represent the case where only one study has a null effect. The approach avoids the double dichotomization for significance of the two‐trials rule and has larger project power to detect existing effects over both studies in combination. It can also be used for power calculations and requires a smaller replication sample size than the two‐trials rule for already convincing original studies. We illustrate the performance of the proposed methodology in an application to data from the Experimental Economics Replication Project.

Suggested Citation

  • Charlotte Micheloud & Fadoua Balabdaoui & Leonhard Held, 2023. "Assessing replicability with the sceptical p$$ p $$‐value: Type‐I error control and sample size planning," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 77(4), pages 573-591, November.
    • Handle: RePEc:bla:stanee:v:77:y:2023:i:4:p:573-591
    DOI: 10.1111/stan.12312
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/stan.12312
    Download Restriction: no
    File URL: https://libkey.io/10.1111/stan.12312?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Attila Ambrus & Ben Greiner, 2012. "Imperfect Public Monitoring with Costly Punishment: An Experimental Study," American Economic Review, American Economic Association, vol. 102(7), pages 3317-3332, December.
    2. Leonhard Held, 2020. "A new standard for the analysis and design of replication studies," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 183(2), pages 431-448, February.
    3. Samuel Pawel & Leonhard Held, 2022. "The sceptical Bayes factor for the assessment of replication success," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 84(3), pages 879-911, July.
    4. Samuel Pawel & Leonhard Held, 2020. "Probabilistic forecasting of replication studies," PLOS ONE, Public Library of Science, vol. 15(4), pages 1-23, April.
    5. Keith M. Marzilli Ericson & Andreas Fuster, 2011. "Expectations as Endowments: Evidence on Reference-Dependent Preferences from Exchange and Valuation Experiments," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(4), pages 1879-1907.
    6. Alexander Shapiro & Jos Berge, 2002. "Statistical inference of minimum rank factor analysis," Psychometrika, Springer;The Psychometric Society, vol. 67(1), pages 79-94, March.
    7. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    8. D. A. S. Fraser, 2019. "The p-value Function and Statistical Inference," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 135-147, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. repec:osf:metaar:apdxk_v1 is not listed on IDEAS

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Konstantinos Bourazas & Guido Consonni & Laura Deldossi, 2024. "Bayesian sample size determination for detecting heterogeneity in multi-site replication studies," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 33(3), pages 697-716, September.
    2. Samuel Pawel & Frederik Aust & Leonhard Held & Eric-Jan Wagenmakers, 2024. "Power priors for replication studies," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 33(1), pages 127-154, March.
    3. Holger Herz & Dmitry Taubinsky, 2018. "What Makes a Price Fair? An Experimental Study of Transaction Experience and Endogenous Fairness Views," Journal of the European Economic Association, European Economic Association, vol. 16(2), pages 316-352.
    4. Rutledge, Robert M. & Alladi, Vinayak & Cheung, Stephen L., 2025. "Price expectations and reference-dependent preferences," Journal of Economic Behavior & Organization, Elsevier, vol. 231(C).
    5. Simon Gächter & Lingbo Huang & Martin Sefton, 2018. "Disappointment Aversion And Social Comparisons In A Real‐Effort Competition," Economic Inquiry, Western Economic Association International, vol. 56(3), pages 1512-1525, July.
    6. Samuel Pawel & Leonhard Held, 2022. "The sceptical Bayes factor for the assessment of replication success," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 84(3), pages 879-911, July.
    7. Nicolas Vallois & Dorian Jullien, 2017. "Replication in Experimental Economics: A Historical and Quantitative Approach Focused on Public Good Game Experiments," GREDEG Working Papers 2017-21, Groupe de REcherche en Droit, Economie, Gestion (GREDEG CNRS), Université Côte d'Azur, France.
    8. Nicolas Vallois & Dorian Jullien, 2017. "Replication in experimental economics: A historical and quantitative approach focused on public good game experiments," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-01651080, HAL.
    9. Heffetz, Ori, 2021. "Are reference points merely lagged beliefs over probabilities?," Journal of Economic Behavior & Organization, Elsevier, vol. 181(C), pages 252-269.
    10. Muradchanian, Jasmine & Hoekstra, Rink & Kiers, Henk & van Ravenzwaaij, Don, 2020. "How Best to Quantify Replication Success? A Simulation Study on the Comparison of Replication Success Metrics," MetaArXiv wvdjf, Center for Open Science.
    11. Lawrence L. Kupper & Sandra L. Martin, 2022. "Replication study design: confidence intervals and commentary," Statistical Papers, Springer, vol. 63(5), pages 1577-1583, October.
    12. Fabrice Lec & Serge Macé, 2018. "The curse of hope," Theory and Decision, Springer, vol. 84(3), pages 429-451, May.
    13. Holger Herz & Dmitry Taubinsky, 2016. "What Makes a Price Fair? An Experimental Analysis of Transaction Experience and Endogenous Fairness Views," NBER Working Papers 22728, National Bureau of Economic Research, Inc.
    14. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    15. repec:osf:metaar:apdxk_v1 is not listed on IDEAS
    16. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 48(1), pages 62-83.
    17. Marcela Ibanez & Sebastian O. Schneider, 2023. "Income Risk, Precautionary Saving, and Loss Aversion – An Empirical Test," Discussion Paper Series of the Max Planck Institute for Behavioral Economics 2023_06, Max Planck Institute for Behavioral Economics.
    18. Nowak, Piotr Bolesław, 2016. "The MLE of the mean of the exponential distribution based on grouped data is stochastically increasing," Statistics & Probability Letters, Elsevier, vol. 111(C), pages 49-54.
    19. Hu, Li & Ma, Hoi-Lam & Wang, Li & Liu, Yang, 2023. "Hiding or disclosing? Information discrimination in member-only discounts," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 171(C).
    20. Dirk Tasche, 2009. "Estimating discriminatory power and PD curves when the number of defaults is small," Papers 0905.3928, arXiv.org, revised Mar 2010.
    21. Karim, Md Aktar Ul & Bhagat, Supriya Ramdas & Bhowmick, Amiya Ranjan, 2022. "Empirical detection of parameter variation in growth curve models using interval specific estimators," Chaos, Solitons & Fractals, Elsevier, vol. 157(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:stanee:v:77:y:2023:i:4:p:573-591. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.blackwellpublishing.com/journal.asp?ref=0039-0402 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.