IDEAS home Printed from https://ideas.repec.org/a/spr/stpapr/v63y2022i5d10.1007_s00362-022-01291-2.html
   My bibliography  Save this article

Replication study design: confidence intervals and commentary

Author

Listed:
  • Lawrence L. Kupper

    (University of North Carolina)

  • Sandra L. Martin

    (University of North Carolina)

Abstract

Methods for designing a comparable replication study have received considerable attention in the published literature, with both Bayesian and non-Bayesian methods having been developed from a hypothesis testing and associated P-value perspective. The purpose of this paper is to describe, using a maximum likelihood-based confidence interval framework, a new frequentist method for choosing the sample size for a comparable replication study. This new method is compared to the published “predictive power” (or “PP”) method. For each of these two methods, a new and easy-to-use formula is derived for computing the optimal comparable replication study sample size that guarantees satisfying a specific confidence interval criterion with a chosen high minimum probability. Connections to hypothesis testing are made, and the Discussion section provides further commentary and considers a numerical example involving published data.

Suggested Citation

  • Lawrence L. Kupper & Sandra L. Martin, 2022. "Replication study design: confidence intervals and commentary," Statistical Papers, Springer, vol. 63(5), pages 1577-1583, October.
    • Handle: RePEc:spr:stpapr:v:63:y:2022:i:5:d:10.1007_s00362-022-01291-2
    DOI: 10.1007/s00362-022-01291-2
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s00362-022-01291-2
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s00362-022-01291-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Leonhard Held, 2020. "A new standard for the analysis and design of replication studies," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 183(2), pages 431-448, February.
    2. Johannes Abeler & Armin Falk & Lorenz Goette & David Huffman, 2011. "Reference Points and Effort Provision," American Economic Review, American Economic Association, vol. 101(2), pages 470-492, April.
    3. Ronald L. Wasserstein & Nicole A. Lazar, 2016. "The ASA's Statement on p -Values: Context, Process, and Purpose," The American Statistician, Taylor & Francis Journals, vol. 70(2), pages 129-133, May.
    4. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    2. Oliver Schilke & Sheen S. Levine & Olenka Kacperczyk & Lynne G. Zucker, 2019. "Call for Papers-Special Issue on Experiments in Organizational Theory," Organization Science, INFORMS, vol. 30(1), pages 232-234, February.
    3. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    4. Fabrice Lec & Serge Macé, 2018. "The curse of hope," Theory and Decision, Springer, vol. 84(3), pages 429-451, May.
    5. Anderson, Brian S. & Wennberg, Karl & McMullen, Jeffery S., 2019. "Editorial: Enhancing quantitative theory-testing entrepreneurship research," Journal of Business Venturing, Elsevier, vol. 34(5), pages 1-1.
    6. Muradchanian, Jasmine & Hoekstra, Rink & Kiers, Henk & van Ravenzwaaij, Don, 2020. "How Best to Quantify Replication Success? A Simulation Study on the Comparison of Replication Success Metrics," MetaArXiv wvdjf, Center for Open Science.
    7. Simon Gächter & Lingbo Huang & Martin Sefton, 2018. "Disappointment Aversion And Social Comparisons In A Real‐Effort Competition," Economic Inquiry, Western Economic Association International, vol. 56(3), pages 1512-1525, July.
    8. Samuel Pawel & Leonhard Held, 2022. "The sceptical Bayes factor for the assessment of replication success," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 84(3), pages 879-911, July.
    9. Heffetz, Ori, 2021. "Are reference points merely lagged beliefs over probabilities?," Journal of Economic Behavior & Organization, Elsevier, vol. 181(C), pages 252-269.
    10. Bruns, Stephan B. & Asanov, Igor & Bode, Rasmus & Dunger, Melanie & Funk, Christoph & Hassan, Sherif M. & Hauschildt, Julia & Heinisch, Dominik & Kempa, Karol & König, Johannes & Lips, Johannes & Verb, 2019. "Reporting errors and biases in published empirical findings: Evidence from innovation research," Research Policy, Elsevier, vol. 48(9), pages 1-1.
    11. Holger Herz & Dmitry Taubinsky, 2016. "What Makes a Price Fair? An Experimental Analysis of Transaction Experience and Endogenous Fairness Views," NBER Working Papers 22728, National Bureau of Economic Research, Inc.
    12. Beau Coker & Cynthia Rudin & Gary King, 2021. "A Theory of Statistical Inference for Ensuring the Robustness of Scientific Results," Management Science, INFORMS, vol. 67(10), pages 6174-6197, October.
    13. Doran, Kirk, 2014. "Are long-term wage elasticities of labor supply more negative than short-term ones?," Economics Letters, Elsevier, vol. 122(2), pages 208-210.
    14. da Silveira, Jaylson Jair & Lima, Gilberto Tadeu, 2021. "Wage inequality as a source of endogenous macroeconomic fluctuations," Structural Change and Economic Dynamics, Elsevier, vol. 56(C), pages 35-52.
    15. Ispano, Alessandro & Schwardmann, Peter, 2017. "Cooperating over losses and competing over gains: A social dilemma experiment," Games and Economic Behavior, Elsevier, vol. 105(C), pages 329-348.
    16. Jean-Michel Benkert, 2015. "Bilateral trade with loss-averse agents," ECON - Working Papers 188, Department of Economics - University of Zurich, revised Jul 2022.
    17. Alexander Frankel & Maximilian Kasy, 2022. "Which Findings Should Be Published?," American Economic Journal: Microeconomics, American Economic Association, vol. 14(1), pages 1-38, February.
    18. Karle, Heiko & Schumacher, Heiner & Vølund, Rune, 2023. "Consumer loss aversion and scale-dependent psychological switching costs," Games and Economic Behavior, Elsevier, vol. 138(C), pages 214-237.
    19. Jyotirmoy Sarkar, 2018. "Will P†Value Triumph over Abuses and Attacks?," Biostatistics and Biometrics Open Access Journal, Juniper Publishers Inc., vol. 7(4), pages 66-71, July.
    20. Celse, Jeremy & Karakostas, Alexandros & Zizzo, Daniel John, 2023. "Relative risk taking and social curiosity," Journal of Economic Behavior & Organization, Elsevier, vol. 210(C), pages 243-264.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:stpapr:v:63:y:2022:i:5:d:10.1007_s00362-022-01291-2. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.