IDEAS home Printed from https://ideas.repec.org/p/lvl/lacicr/1425.html
   My bibliography  Save this paper

Statistical Power of Within and Between-Subjects Designs in Economic Experiments

Author

Listed:
  • Charles Bellemare
  • Luc Bissonnette
  • Sabine Kröger

Abstract

This paper discusses the choice of the number of participants for within-subjects (WS) designs and between-subjects (BS) designs based on simulations of statistical power allowing for different numbers of experimental periods. We illustrate the usefulness of the approach in the context of field experiments on gift exchange. Our results suggest that a BS design requires between 4 to 8 times more subjects than a WS design to reach an acceptable level of statistical power. Moreover, the predicted minimal sample sizes required to correctly detect a treatment effect with a probability of 80% greatly exceed sizes currently used in the literature. Our results suggest that adding experimental periods in an experiment can substantially increase the statistical power of a WS design, but have very little effect on the statistical power of the BS design. Finally, we discuss issues relating to numerical computation and present the powerBBK package programmed for STATA. This package allows users to conduct their own analysis of power for the different designs (WS and BS), conditional on user specified experimental parameters (true effect size, sample size, number of periods, noise levels for control and treatment, error distributions), statistical tests (parametric and nonparametric), and estimation methods (linear regression, binary choice models (probit and logit), censored regression models (tobit)).

Suggested Citation

  • Charles Bellemare & Luc Bissonnette & Sabine Kröger, 2014. "Statistical Power of Within and Between-Subjects Designs in Economic Experiments," Cahiers de recherche 1425, CIRPEE.
    • Handle: RePEc:lvl:lacicr:1425
    as

    Download full text from publisher

    File URL: http://www.cirpee.org/fileadmin/documents/Cahiers_2014/CIRPEE14-25.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Jan Stoop, 2014. "From the lab to the field: envelopes, dictators and manners," Experimental Economics, Springer;Economic Science Association, vol. 17(2), pages 304-313, June.
    2. Rutström, E. Elisabet & Wilcox, Nathaniel T., 2009. "Stated beliefs versus inferred beliefs: A methodological inquiry and experimental test," Games and Economic Behavior, Elsevier, vol. 67(2), pages 616-632, November.
    3. A. H. Feiveson, 2002. "Power by simulation," Stata Journal, StataCorp LP, vol. 2(2), pages 107-124, May.
    4. Peter Bossaerts & Charles Plott & William R. Zame, 2007. "Prices and Portfolio Choices in Financial Markets: Theory, Econometrics, Experiments," Econometrica, Econometric Society, vol. 75(4), pages 993-1038, July.
    5. John List & Sally Sadoff & Mathis Wagner, 2011. "So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design," Experimental Economics, Springer;Economic Science Association, vol. 14(4), pages 439-457, November.
    6. Charness, Gary & Gneezy, Uri & Kuhn, Michael A., 2012. "Experimental methods: Between-subject and within-subject design," Journal of Economic Behavior & Organization, Elsevier, vol. 81(1), pages 1-8.
    7. Trautmann, Stefan T. & van de Kuilen, Gijs & Zeckhauser, Richard J., 2013. "Social Class and Un(ethical) Behavior: A Framework, with Evidence from a Large Population Sample," Working Paper Series rwp13-004, Harvard University, John F. Kennedy School of Government.
    8. Bellemare, Charles & Shearer, Bruce, 2009. "Gift giving and worker productivity: Evidence from a firm-level experiment," Games and Economic Behavior, Elsevier, vol. 67(1), pages 233-244, September.
    9. Andreas Ortman & Le Zhang, 2013. "Exploring the Meaning of Significance in Experimental Economics," Discussion Papers 2013-32, School of Economics, The University of New South Wales.
    10. De Long, J Bradford & Lang, Kevin, 1992. "Are All Economic Hypotheses False?," Journal of Political Economy, University of Chicago Press, vol. 100(6), pages 1257-1272, December.
    11. Paul J. Ferraro & Michael K. Price, 2013. "Using Nonpecuniary Strategies to Influence Behavior: Evidence from a Large-Scale Field Experiment," The Review of Economics and Statistics, MIT Press, vol. 95(1), pages 64-73, March.
    12. Smith, Vernon L, et al, 1982. "Competitive Market Institutions: Double Auctions vs. Sealed Bid-Offer Auctions," American Economic Review, American Economic Association, vol. 72(1), pages 58-77, March.
    13. Uri Gneezy & John A List, 2006. "Putting Behavioral Economics to Work: Testing for Gift Exchange in Labor Markets Using Field Experiments," Econometrica, Econometric Society, vol. 74(5), pages 1365-1384, September.
    14. Maarten J. Voors & Eleonora E. M. Nillesen & Philip Verwimp & Erwin H. Bulte & Robert Lensink & Daan P. Van Soest, 2012. "Violent Conflict and Behavior: A Field Experiment in Burundi," American Economic Review, American Economic Association, vol. 102(2), pages 941-964, April.
    15. McKenzie, David, 2012. "Beyond baseline and follow-up: The case for more T in experiments," Journal of Development Economics, Elsevier, vol. 99(2), pages 210-221.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Igor Asanov & Christoph Buehren & Panagiota Zacharodimou, 2020. "The power of experiments: How big is your n?," MAGKS Papers on Economics 202032, Philipps-Universität Marburg, Faculty of Business Administration and Economics, Department of Economics (Volkswirtschaftliche Abteilung).
    2. Reddinger, J. Lucas & Charness, Gary & Levine, David, 2022. "Prosocial motivation for vaccination," SocArXiv emj6v, Center for Open Science.
    3. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    4. Kathryn N. Vasilaky & J. Michelle Brock, 2020. "Power(ful) guidelines for experimental economists," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(2), pages 189-212, December.
    5. Iavor Bojinov & Ashesh Rambachan & Neil Shephard, 2021. "Panel experiments and dynamic causal effects: A finite population perspective," Quantitative Economics, Econometric Society, vol. 12(4), pages 1171-1196, November.
    6. Cloos, Janis & Greiff, Matthias & Rusch, Hannes, 2019. "Geographical Concentration and Editorial Favoritism within the Field of Laboratory Experimental Economics," Research Memorandum 029, Maastricht University, Graduate School of Business and Economics (GSBE).
    7. Cloos, Janis & Greiff, Matthias & Rusch, Hannes, 2020. "Geographical Concentration and Editorial Favoritism within the Field of Laboratory Experimental Economics (RM/19/029-revised-)," Research Memorandum 014, Maastricht University, Graduate School of Business and Economics (GSBE).
    8. Braaten, Ragnhild Haugli & Brekke, Kjell Arne & Rogeberg, Ole, 2015. "Buying the right to do wrong – An experimental test of moral objections to trading emission permits," Resource and Energy Economics, Elsevier, vol. 42(C), pages 110-124.
    9. Lennart Erixon & Louise Johannesson, 2015. "Is the psychology of high profits detrimental to industrial renewal? Experimental evidence for the theory of transformation pressure," Journal of Evolutionary Economics, Springer, vol. 25(2), pages 475-511, April.
    10. Cloos, Janis & Greiff, Matthias & Rusch, Hannes, 2021. "Editorial favoritism in the field of laboratory experimental economics (RM/20/014-revised-)," Research Memorandum 005, Maastricht University, Graduate School of Business and Economics (GSBE).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    2. Leah H. Palm-Forster & Paul J. Ferraro & Nicholas Janusch & Christian A. Vossler & Kent D. Messer, 2019. "Behavioral and Experimental Agri-Environmental Research: Methodological Challenges, Literature Gaps, and Recommendations," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 73(3), pages 719-742, July.
    3. Charles Bellemare & Luc Bissonnette & Sabine Kröger, 2016. "Simulating power of economic experiments: the powerBBK package," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 2(2), pages 157-168, November.
    4. Alain Cohn & Ernst Fehr & Lorenz Goette, 2015. "Fair Wages and Effort Provision: Combining Evidence from a Choice Experiment and a Field Experiment," Management Science, INFORMS, vol. 61(8), pages 1777-1794, August.
    5. Brian Albert Monroe, 2020. "The statistical power of individual-level risk preference estimation," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(2), pages 168-188, December.
    6. Kathryn N. Vasilaky & J. Michelle Brock, 2020. "Power(ful) guidelines for experimental economists," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(2), pages 189-212, December.
    7. Brendon McConnell & Marcos Vera-Hernandez, 2015. "Going beyond simple sample size calculations: a practitioner's guide," IFS Working Papers W15/17, Institute for Fiscal Studies.
    8. Zacharias Maniadis & Fabio Tufano & John A. List, 2017. "To Replicate or Not To Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study," Economic Journal, Royal Economic Society, vol. 127(605), pages 209-235, October.
    9. da Silveira, Jaylson Jair & Lima, Gilberto Tadeu, 2021. "Wage inequality as a source of endogenous macroeconomic fluctuations," Structural Change and Economic Dynamics, Elsevier, vol. 56(C), pages 35-52.
    10. Pedro Carneiro & Sokbae Lee & Daniel Wilhelm, 2020. "Optimal data collection for randomized control trials [Microcredit impacts: Evidence from a randomized microcredit program placement experiment by Compartamos Banco]," The Econometrics Journal, Royal Economic Society, vol. 23(1), pages 1-31.
    11. Faralla, Valeria & Borà, Guido & Innocenti, Alessandro & Novarese, Marco, 2020. "Promises in group decision making," Research in Economics, Elsevier, vol. 74(1), pages 1-11.
    12. Elliott Ash & W. Bentley MacLeod, 2015. "Intrinsic Motivation in Public Service: Theory and Evidence from State Supreme Courts," Journal of Law and Economics, University of Chicago Press, vol. 58(4).
    13. Omar Al-Ubaydli & John A. List, 2019. "How natural field experiments have enhanced our understanding of unemployment," Nature Human Behaviour, Nature, vol. 3(1), pages 33-39, January.
    14. Sliwka, Dirk & Werner, Peter, 2016. "How Do Agents React to Dynamic Wage Increases? An Experimental Study," IZA Discussion Papers 9855, Institute of Labor Economics (IZA).
    15. ITO Koichiro & IDA Takanori & TANAKA Makoto, 2015. "The Persistence of Moral Suasion and Economic Incentives: Field experimental evidence from energy demand," Discussion papers 15014, Research Institute of Economy, Trade and Industry (RIETI).
    16. Craig Landry & Andreas Lange & John List & Michael Price & Nicholas Rupp, 2011. "Is There a 'Hidden Cost of Control' in Naturally-Occurring Markets? Evidence from a Natural Field Experiment," Natural Field Experiments 00593, The Field Experiments Website.
    17. Bartoš, Vojtěch, 2021. "Seasonal scarcity and sharing norms," Journal of Economic Behavior & Organization, Elsevier, vol. 185(C), pages 303-316.
    18. Bogliacino, Francesco & Grimalda, Gianluca & Pipke, David, 2021. "Kind or contented? An investigation of the gift exchange hypothesis in a natural field experiment in Colombia," OSF Preprints xmjaq, Center for Open Science.
    19. Omar Al-Ubaydli & John List, 2013. "On the Generalizability of Experimental Results in Economics: With A Response To Camerer," Artefactual Field Experiments j0001, The Field Experiments Website.
    20. Regner, Tobias, 2014. "Social preferences? Google Answers!," Games and Economic Behavior, Elsevier, vol. 85(C), pages 188-209.

    More about this item

    Keywords

    Within-subjects design; Between-subjects design; sample sizes; statistical power; experiments;
    All these keywords.

    JEL classification:

    • C8 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs
    • C9 - Mathematical and Quantitative Methods - - Design of Experiments
    • D03 - Microeconomics - - General - - - Behavioral Microeconomics: Underlying Principles

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:lvl:lacicr:1425. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Manuel Paradis (email available below). General contact details of provider: https://edirc.repec.org/data/cirpeca.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.