IDEAS home Printed from https://ideas.repec.org/a/wly/fistud/v46y2025i3p323-348.html

Going beyond simple sample size calculations: a practitioner's guide

Author

Listed:
  • Brendon McConnell
  • Marcos Vera‐Hernández

Abstract

Basic methods to compute required sample sizes are well understood and supported by widely available software. However, researchers often oversimplify their sample size calculations, overlooking relevant features of their experimental design. This paper compiles and systematises existing methods for sample size calculations for continuous and binary outcomes, both with and without covariates, and for both clustered and non‐clustered randomised controlled trials. We present formulae accommodating panel data structures and uneven designs, and provide guidance on optimally allocating sample size between the number of clusters and the number of units per cluster. In addition, we discuss how to adjust calculations for multiple hypothesis testing and how to estimate power in more complex designs using simulation methods.

Suggested Citation

  • Brendon McConnell & Marcos Vera‐Hernández, 2025. "Going beyond simple sample size calculations: a practitioner's guide," Fiscal Studies, John Wiley & Sons, vol. 46(3), pages 323-348, September.
    • Handle: RePEc:wly:fistud:v:46:y:2025:i:3:p:323-348
    DOI: 10.1111/1475-5890.70005
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/1475-5890.70005
    Download Restriction: no
    File URL: https://libkey.io/10.1111/1475-5890.70005?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Anderson, Michael L., 2008. "Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects," Journal of the American Statistical Association, American Statistical Association, vol. 103(484), pages 1481-1495.
    2. Kenneth F Schulz & Douglas G Altman & David Moher & for the CONSORT Group, 2010. "CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials," PLOS Medicine, Public Library of Science, vol. 7(3), pages 1-7, March.
    3. repec:mpr:mprres:7874 is not listed on IDEAS
    4. Richard Blundell & Monica Costa Dias, 2009. "Alternative Approaches to Evaluation in Empirical Microeconomics," Journal of Human Resources, University of Wisconsin Press, vol. 44(3).
    5. Richard Hooper, 2013. "Versatile sample-size calculation using simulation," Stata Journal, StataCorp LLC, vol. 13(1), pages 21-38, March.
    6. repec:feb:artefa:0087 is not listed on IDEAS
    7. A. Colin Cameron & Jonah B. Gelbach & Douglas L. Miller, 2008. "Bootstrap-Based Improvements for Inference with Clustered Errors," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 414-427, August.
    8. James J. Heckman & Jeffrey A. Smith, 1995. "Assessing the Case for Social Experiments," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 85-110, Spring.
    9. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    10. David McKenzie, 2025. "Designing and analysing powerful experiments: practical tips for applied researchers," Fiscal Studies, John Wiley & Sons, vol. 46(3), pages 305-322, September.
    11. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Cluster-robust inference: A guide to empirical practice," Journal of Econometrics, Elsevier, vol. 232(2), pages 272-299.
    12. McKenzie, David, 2012. "Beyond baseline and follow-up: The case for more T in experiments," Journal of Development Economics, Elsevier, vol. 99(2), pages 210-221.
    13. Levitt, Steven D. & List, John A., 2009. "Field experiments in economics: The past, the present, and the future," European Economic Review, Elsevier, vol. 53(1), pages 1-18, January.
    14. Jerry A. Hausman & David A. Wise, 1985. "Introduction to "Social Experimentation"," NBER Chapters, in: Social Experimentation, pages 1-10, National Bureau of Economic Research, Inc.
    15. Hausman, Jerry A. & Wise, David A. (ed.), 1985. "Social Experimentation," National Bureau of Economic Research Books, University of Chicago Press, number 9780226319407, March.
    16. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    17. repec:mpr:mprres:6371 is not listed on IDEAS
    18. Jerry A. Hausman & David A. Wise, 1985. "Social Experimentation," NBER Books, National Bureau of Economic Research, Inc, number haus85-1.
    19. John List & Sally Sadoff & Mathis Wagner, 2011. "So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design," Experimental Economics, Springer;Economic Science Association, vol. 14(4), pages 439-457, November.
    20. Kontopantelis, Evangelos & Springate, David A & Parisi, Rosa & Reeves, David, 2016. "Simulation-Based Power Calculations for Mixed Effects Modeling: ipdpower in Stata," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 74(i12).
    21. Gary Burtless, 1995. "The Case for Randomized Field Trials in Economic and Policy Research," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 63-84, Spring.
    22. Manoj Mohanan & Katherine Donato & Grant Miller & Yulya Truskinovsky & Marcos Vera-Hernández, 2021. "Different Strokes for Different Folks? Experimental Evidence on the Effectiveness of Input and Output Incentive Contracts for Health Care Providers with Varying Skills," American Economic Journal: Applied Economics, American Economic Association, vol. 13(4), pages 34-69, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. David McKenzie, 2025. "Designing and analysing powerful experiments: practical tips for applied researchers," Fiscal Studies, John Wiley & Sons, vol. 46(3), pages 305-322, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Brendon McConnell & Marcos Vera-Hernandez, 2015. "Going beyond simple sample size calculations: a practitioner's guide," IFS Working Papers W15/17, Institute for Fiscal Studies.
    2. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    3. Sant’Anna, Pedro H.C. & Zhao, Jun, 2020. "Doubly robust difference-in-differences estimators," Journal of Econometrics, Elsevier, vol. 219(1), pages 101-122.
    4. Eric Floyd & John A. List, 2016. "Using Field Experiments in Accounting and Finance," Journal of Accounting Research, John Wiley & Sons, Ltd., vol. 54(2), pages 437-475, May.
    5. John V. Pepper, 2003. "Using Experiments to Evaluate Performance Standards: What Do Welfare-to-Work Demonstrations Reveal to Welfare Reformers?," Journal of Human Resources, University of Wisconsin Press, vol. 38(4).
    6. Brown, Annette N. & Wood, Benjamin Douglas Kuflick, 2018. "Which tests not witch hunts: A diagnostic approach for conducting replication research," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy, vol. 12, pages 1-26.
    7. Schneider, Hilmar & Uhlendorff, Arne & Zimmermann, Klaus F., 2010. "Mit Workfare aus der Sozialhilfe? Lehren aus einem Modellprojekt," IZA Standpunkte 33, Institute of Labor Economics (IZA).
    8. Gustavo Canavire-Bacarreza & Luis Castro Peñarrieta & Darwin Ugarte Ontiveros, 2021. "Outliers in Semi-Parametric Estimation of Treatment Effects," Econometrics, MDPI, vol. 9(2), pages 1-32, April.
    9. Levitt, Steven D. & List, John A., 2009. "Field experiments in economics: The past, the present, and the future," European Economic Review, Elsevier, vol. 53(1), pages 1-18, January.
    10. Bandiera, Oriana. & Buehren, Niklas. & Burgess, Robin & Goldstein, Markus P., & Gulesci, Selim. & Rasul, Imran. & Sulaiman, Munshi., 2015. "Women’s economic empowerment in action : evidence from a randomized control trial in Africa," ILO Working Papers 994874053402676, International Labour Organization.
    11. Marshall Burke & Lauren Falcao Bergquist & Edward Miguel, 2019. "Sell Low and Buy High: Arbitrage and Local Price Effects in Kenyan Markets," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 134(2), pages 785-842.
    12. Angus Deaton, 2010. "Instruments, Randomization, and Learning about Development," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 424-455, June.
    13. Abhijit Banerjee & Rukmini Banerji & James Berry & Esther Duflo & Harini Kannan & Shobhini Mukerji & Marc Shotland & Michael Walton, 2017. "From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 73-102, Fall.
    14. Omar Al-Ubaydli & John List, 2013. "On the Generalizability of Experimental Results in Economics: With A Response To Camerer," Artefactual Field Experiments j0001, The Field Experiments Website.
    15. Francesco Cecchi & Steffen Eriksen & Robert Lensink & Paul Mosley, 2025. "Is Microcredit Effective? It Depends on the Context: New Results from a Study in Bolivia," The European Journal of Development Research, Palgrave Macmillan;European Association of Development Research and Training Institutes (EADI), vol. 37(3), pages 593-642, June.
    16. Nobel Prize Committee, 2019. "Understanding development and poverty alleviation," Nobel Prize in Economics documents 2019-2, Nobel Prize Committee.
    17. Hunt Allcott, 2012. "Site Selection Bias in Program Evaluation," NBER Working Papers 18373, National Bureau of Economic Research, Inc.
    18. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    19. Arun Advani & Tymon Słoczyński, 2013. "Mostly harmless simulations? On the internal validity of empirical Monte Carlo studies," CeMMAP working papers 64/13, Institute for Fiscal Studies.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:fistud:v:46:y:2025:i:3:p:323-348. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1111/(ISSN)1475-5890 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.