IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp8583.html
   My bibliography  Save this paper

Statistical Power of Within and Between-Subjects Designs in Economic Experiments

Author

Listed:
  • Bellemare, Charles

    () (Université Laval)

  • Bissonnette, Luc

    () (Laval University)

  • Kröger, Sabine

    () (Université Laval)

Abstract

This paper discusses the choice of the number of participants for within-subjects (WS) designs and between-subjects (BS) designs based on simulations of statistical power allowing for different numbers of experimental periods. We illustrate the usefulness of the approach in the context of field experiments on gift exchange. Our results suggest that a BS design requires between 4 to 8 times more subjects than a WS design to reach an acceptable level of statistical power. Moreover, the predicted minimal sample sizes required to correctly detect a treatment effect with a probability of 80% greatly exceed sizes currently used in the literature. Our results suggest that adding experimental periods in an experiment can substantially increase the statistical power of a WS design, but have very little effect on the statistical power of the BS design. Finally, we discuss issues relating to numerical computation and present the powerBBK package programmed for STATA. This package allows users to conduct their own analysis of power for the different designs (WS and BS), conditional on user specified experimental parameters (true effect size, sample size, number of periods, noise levels for control and treatment, error distributions), statistical tests (parametric and nonparametric), and estimation methods (linear regression, binary choice models (probit and logit), censored regression models (tobit)).

Suggested Citation

  • Bellemare, Charles & Bissonnette, Luc & Kröger, Sabine, 2014. "Statistical Power of Within and Between-Subjects Designs in Economic Experiments," IZA Discussion Papers 8583, Institute of Labor Economics (IZA).
  • Handle: RePEc:iza:izadps:dp8583
    as

    Download full text from publisher

    File URL: http://ftp.iza.org/dp8583.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Jan Stoop, 2014. "From the lab to the field: envelopes, dictators and manners," Experimental Economics, Springer;Economic Science Association, vol. 17(2), pages 304-313, June.
    2. Rutström, E. Elisabet & Wilcox, Nathaniel T., 2009. "Stated beliefs versus inferred beliefs: A methodological inquiry and experimental test," Games and Economic Behavior, Elsevier, vol. 67(2), pages 616-632, November.
    3. A. H. Feiveson, 2002. "Power by simulation," Stata Journal, StataCorp LP, vol. 2(2), pages 107-124, May.
    4. Peter Bossaerts & Charles Plott & William R. Zame, 2007. "Prices and Portfolio Choices in Financial Markets: Theory, Econometrics, Experiments," Econometrica, Econometric Society, vol. 75(4), pages 993-1038, July.
    5. John List & Sally Sadoff & Mathis Wagner, 2011. "So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design," Experimental Economics, Springer;Economic Science Association, vol. 14(4), pages 439-457, November.
    6. Charness, Gary & Gneezy, Uri & Kuhn, Michael A., 2012. "Experimental methods: Between-subject and within-subject design," Journal of Economic Behavior & Organization, Elsevier, vol. 81(1), pages 1-8.
    7. Bellemare, Charles & Shearer, Bruce, 2009. "Gift giving and worker productivity: Evidence from a firm-level experiment," Games and Economic Behavior, Elsevier, vol. 67(1), pages 233-244, September.
    8. De Long, J Bradford & Lang, Kevin, 1992. "Are All Economic Hypotheses False?," Journal of Political Economy, University of Chicago Press, vol. 100(6), pages 1257-1272, December.
    9. Feiveson, A.H., 2002. "Power by simulation," Stata Journal, StataCorp LP, vol. 2(2), pages 1-18.
    10. Paul J. Ferraro & Michael K. Price, 2013. "Using Nonpecuniary Strategies to Influence Behavior: Evidence from a Large-Scale Field Experiment," The Review of Economics and Statistics, MIT Press, vol. 95(1), pages 64-73, March.
    11. Smith, Vernon L, et al, 1982. "Competitive Market Institutions: Double Auctions vs. Sealed Bid-Offer Auctions," American Economic Review, American Economic Association, vol. 72(1), pages 58-77, March.
    12. Maarten J. Voors & Eleonora E. M. Nillesen & Philip Verwimp & Erwin H. Bulte & Robert Lensink & Daan P. Van Soest, 2012. "Violent Conflict and Behavior: A Field Experiment in Burundi," American Economic Review, American Economic Association, vol. 102(2), pages 941-964, April.
    13. McKenzie, David, 2012. "Beyond baseline and follow-up: The case for more T in experiments," Journal of Development Economics, Elsevier, vol. 99(2), pages 210-221.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Braaten, Ragnhild Haugli & Brekke, Kjell Arne & Rogeberg, Ole, 2015. "Buying the right to do wrong – An experimental test of moral objections to trading emission permits," Resource and Energy Economics, Elsevier, vol. 42(C), pages 110-124.
    2. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    3. Lennart Erixon & Louise Johannesson, 2015. "Is the psychology of high profits detrimental to industrial renewal? Experimental evidence for the theory of transformation pressure," Journal of Evolutionary Economics, Springer, vol. 25(2), pages 475-511, April.

    More about this item

    Keywords

    within-subjects design; between-subjects design; sample sizes; statistical power; experiments;

    JEL classification:

    • C8 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs
    • C9 - Mathematical and Quantitative Methods - - Design of Experiments
    • D03 - Microeconomics - - General - - - Behavioral Microeconomics: Underlying Principles

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp8583. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Holger Hinte). General contact details of provider: http://www.iza.org .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.