IDEAS home Printed from https://ideas.repec.org/a/bla/stanee/v77y2023i4p573-591.html
   My bibliography  Save this article

Assessing replicability with the sceptical p$$ p $$‐value: Type‐I error control and sample size planning

Author

Listed:
  • Charlotte Micheloud
  • Fadoua Balabdaoui
  • Leonhard Held

Abstract

We study a statistical framework for replicability based on a recently proposed quantitative measure of replication success, the sceptical p$$ p $$‐value. A recalibration is proposed to obtain exact overall Type‐I error control if the effect is null in both studies and additional bounds on the partial and conditional Type‐I error rate, which represent the case where only one study has a null effect. The approach avoids the double dichotomization for significance of the two‐trials rule and has larger project power to detect existing effects over both studies in combination. It can also be used for power calculations and requires a smaller replication sample size than the two‐trials rule for already convincing original studies. We illustrate the performance of the proposed methodology in an application to data from the Experimental Economics Replication Project.

Suggested Citation

  • Charlotte Micheloud & Fadoua Balabdaoui & Leonhard Held, 2023. "Assessing replicability with the sceptical p$$ p $$‐value: Type‐I error control and sample size planning," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 77(4), pages 573-591, November.
    • Handle: RePEc:bla:stanee:v:77:y:2023:i:4:p:573-591
    DOI: 10.1111/stan.12312
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/stan.12312
    Download Restriction: no
    File URL: https://libkey.io/10.1111/stan.12312?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Leonhard Held, 2020. "A new standard for the analysis and design of replication studies," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 183(2), pages 431-448, February.
    2. D. A. S. Fraser, 2019. "The p-value Function and Statistical Inference," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 135-147, March.
    3. Attila Ambrus & Ben Greiner, 2012. "Imperfect Public Monitoring with Costly Punishment: An Experimental Study," American Economic Review, American Economic Association, vol. 102(7), pages 3317-3332, December.
    4. Samuel Pawel & Leonhard Held, 2022. "The sceptical Bayes factor for the assessment of replication success," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 84(3), pages 879-911, July.
    5. Samuel Pawel & Leonhard Held, 2020. "Probabilistic forecasting of replication studies," PLOS ONE, Public Library of Science, vol. 15(4), pages 1-23, April.
    6. Alexander Shapiro & Jos Berge, 2002. "Statistical inference of minimum rank factor analysis," Psychometrika, Springer;The Psychometric Society, vol. 67(1), pages 79-94, March.
    7. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    8. Keith M. Marzilli Ericson & Andreas Fuster, 2011. "Expectations as Endowments: Evidence on Reference-Dependent Preferences from Exchange and Valuation Experiments," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(4), pages 1879-1907.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. repec:osf:metaar:apdxk_v1 is not listed on IDEAS

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Konstantinos Bourazas & Guido Consonni & Laura Deldossi, 2024. "Bayesian sample size determination for detecting heterogeneity in multi-site replication studies," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 33(3), pages 697-716, September.
    2. Samuel Pawel & Frederik Aust & Leonhard Held & Eric-Jan Wagenmakers, 2024. "Power priors for replication studies," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 33(1), pages 127-154, March.
    3. Rutledge, Robert M. & Alladi, Vinayak & Cheung, Stephen L., 2025. "Price expectations and reference-dependent preferences," Journal of Economic Behavior & Organization, Elsevier, vol. 231(C).
    4. Nicolas Vallois & Dorian Jullien, 2017. "Replication in experimental economics: A historical and quantitative approach focused on public good game experiments," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-01651080, HAL.
    5. Muradchanian, Jasmine & Hoekstra, Rink & Kiers, Henk & van Ravenzwaaij, Don, 2020. "How Best to Quantify Replication Success? A Simulation Study on the Comparison of Replication Success Metrics," MetaArXiv wvdjf, Center for Open Science.
    6. Simon Gächter & Lingbo Huang & Martin Sefton, 2018. "Disappointment Aversion And Social Comparisons In A Real‐Effort Competition," Economic Inquiry, Western Economic Association International, vol. 56(3), pages 1512-1525, July.
    7. Holger Herz & Dmitry Taubinsky, 2018. "What Makes a Price Fair? An Experimental Study of Transaction Experience and Endogenous Fairness Views," Journal of the European Economic Association, European Economic Association, vol. 16(2), pages 316-352.
    8. Nicolas Vallois & Dorian Jullien, 2017. "Replication in experimental economics: A historical and quantitative approach focused on public good game experiments," Working Papers halshs-01651080, HAL.
    9. Holger Herz & Dmitry Taubinsky, 2016. "What Makes a Price Fair? An Experimental Analysis of Transaction Experience and Endogenous Fairness Views," NBER Working Papers 22728, National Bureau of Economic Research, Inc.
    10. Fabrice Lec & Serge Macé, 2018. "The curse of hope," Theory and Decision, Springer, vol. 84(3), pages 429-451, May.
    11. Lawrence L. Kupper & Sandra L. Martin, 2022. "Replication study design: confidence intervals and commentary," Statistical Papers, Springer, vol. 63(5), pages 1577-1583, October.
    12. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    13. Samuel Pawel & Leonhard Held, 2022. "The sceptical Bayes factor for the assessment of replication success," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 84(3), pages 879-911, July.
    14. Heffetz, Ori, 2021. "Are reference points merely lagged beliefs over probabilities?," Journal of Economic Behavior & Organization, Elsevier, vol. 181(C), pages 252-269.
    15. repec:osf:metaar:apdxk_v1 is not listed on IDEAS
    16. A. Banerji & Neha Gupta, 2011. "Do Auction Bids Betray Expectations-Based Reference Dependent Preferences? A Test, Experimental Evidence, And Estimates Of Loss Aversion," Working papers 206, Centre for Development Economics, Delhi School of Economics.
    17. De Geest, Lawrence R. & Kingsley, David C., 2021. "Norm enforcement with incomplete information," Journal of Economic Behavior & Organization, Elsevier, vol. 189(C), pages 403-430.
    18. Almeida, Sergio, 2023. "Punishment credibility and cooperation in public good games," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 106(C).
    19. Anastasiou, Andreas, 2017. "Bounds for the normal approximation of the maximum likelihood estimator from m-dependent random variables," Statistics & Probability Letters, Elsevier, vol. 129(C), pages 171-181.
    20. Mueller-Langer, Frank & Andreoli-Versbach, Patrick, 2018. "Open access to research data: Strategic delay and the ambiguous welfare effects of mandatory data disclosure," Information Economics and Policy, Elsevier, vol. 42(C), pages 20-34.
    21. Steffen Andersen & Cristian Badarinza & Lu Liu & Julie Marx & Tarun Ramadorai, 2022. "Reference Dependence in the Housing Market," American Economic Review, American Economic Association, vol. 112(10), pages 3398-3440, October.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:stanee:v:77:y:2023:i:4:p:573-591. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.blackwellpublishing.com/journal.asp?ref=0039-0402 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.