IDEAS home Printed from
   My bibliography  Save this article

Randomization inference with Stata: A guide and software


  • Simon Heß

    () (Goethe Universität Frankfurt am Main)


Randomization inference or permutation tests are only sporadically used in economics and other social sciences—this despite a steep increase in ran- domization in field and laboratory experiments that provide perfect experimental setups for applying randomization inference. In the context of causal inference, such tests can handle problems often faced by applied researchers, including issues arising in the context of small samples, stratified or clustered treatment assign- ments, or nonstandard randomization techniques. Standard statistical software packages have either no implementation of randomization tests or very basic im- plementations. Whenever researchers use randomization inference, they regularly code individual program routines, risking inconsistencies and coding mistakes. In this article, I show how randomization inference can best be conducted in Stata and introduce a new command, ritest, to simplify such analyses. I illustrate this approach’s usefulness by replicating the results in Fujiwara and Wantchekon (2013, American Economic Journal: Applied Economics 5: 241–255) and running simulations. The applications cover clustered and stratified assignments, with varying cluster sizes, pairwise randomization, and the computation of nonapprox- imate p-values. The applications also touch upon joint hypothesis testing with randomization inference.

Suggested Citation

  • Simon Heß, 2017. "Randomization inference with Stata: A guide and software," Stata Journal, StataCorp LP, vol. 17(3), pages 630-651, September.
  • Handle: RePEc:tsj:stataj:y:17:y:2017:i:3:p:630-651
    Note: to access software from within Stata, net describe

    Download full text from publisher

    File URL:
    File Function: link to article purchase
    Download Restriction: no

    References listed on IDEAS

    1. Miriam Bruhn & David McKenzie, 2009. "In Pursuit of Balance: Randomization in Practice in Development Field Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 200-232, October.
    2. MacKinnon, James G. & Webb, Matthew D., 2020. "Randomization inference for difference-in-differences with few treated clusters," Journal of Econometrics, Elsevier, vol. 218(2), pages 435-450.
    3. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2018. "Inference Under Covariate-Adaptive Randomization," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(524), pages 1784-1796, October.
    4. Peter Ganong & Simon Jäger, 2018. "A Permutation Test for the Regression Kink Design," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(522), pages 494-504, April.
    5. Jessica Cohen & Pascaline Dupas, 2010. "Free Distribution or Cost-Sharing? Evidence from a Randomized Malaria Prevention Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 125(1), pages 1-45.
    6. Burt S. Barnow & Matias D. Cattaneo & Rocío Titiunik & Gonzalo Vazquez‐Bare, 2017. "Comparing Inference Approaches for RD Designs: A Reexamination of the Effect of Head Start on Child Mortality," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 36(3), pages 643-681, June.
    7. Joseph P. Romano & Michael Wolf, 2005. "Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 94-108, March.
    8. James G. MacKinnon & Matthew D. Webb, 2017. "Wild Bootstrap Inference for Wildly Different Cluster Sizes," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(2), pages 233-254, March.
    9. A. Colin Cameron & Douglas L. Miller, 2015. "A Practitioner’s Guide to Cluster-Robust Inference," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 317-372.
    10. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, December.
    11. Edward E. Leamer, 2010. "Tantalus on the Road to Asymptopia," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 31-46, Spring.
    12. Thomas Fujiwara & Leonard Wantchekon, 2013. "Can Informed Public Deliberation Overcome Clientelism? Experimental Evidence from Benin," American Economic Journal: Applied Economics, American Economic Association, vol. 5(4), pages 241-255, October.
    Full references (including those not matched with items on IDEAS)


    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.

    Cited by:

    1. Kerwin, Jason Theodore & Thornton, Rebecca, 2020. "Making the Grade: The Sensitivity of Education Program Effectiveness to Input Choices and Outcome Measures," SocArXiv ct9sj, Center for Open Science.
    2. Vandevelde, S. & Van Campenhout, B. & Walukano, W., 2018. "Impact of Improving Seed Quality: Evidence from a Video Information Intervention," 2018 Conference, July 28-August 2, 2018, Vancouver, British Columbia 277191, International Association of Agricultural Economists.
    3. Scharfbillig, Mario & Weißler, Marco, 2019. "Heterogeneous displacement effects of migrant labor supply - quasi-experimental evidence from Germany," IAB Discussion Paper 201915, Institut für Arbeitsmarkt- und Berufsforschung (IAB), Nürnberg [Institute for Employment Research, Nuremberg, Germany].
    4. James G. MacKinnon & Matthew D. Webb, 2020. "When and How to Deal with Clustered Errors in Regression Models," Working Paper 1421, Economics Department, Queen's University.
    5. Grimm, Michael & Luck, Nathalie, 2020. "Can Training Enhance Adoption, Knowledge and Perception of Organic Farming Practices? Evidence from a Randomized Experiment in Indonesia," IZA Discussion Papers 13400, Institute of Labor Economics (IZA).
    6. Konstantin Büchel & Martina Jakob & Christoph Kühnhanss & Daniel Steffen & Aymo Brunetti, 2020. "The Relative Effectiveness of Teachers and Learning Software: Evidence from a Field Experiment in El Salvador," University of Bern Social Sciences Working Papers 36, University of Bern, Department of Social Sciences.
    7. Senne Vandevelde & Bjorn Van Campenhout & Wilberforce Walukano, 2018. "Spoiler alert! Spillovers in the context of a video intervention to maintain seed quality among Ugandan potato farmers," LICOS Discussion Papers 40718, LICOS - Centre for Institutions and Economic Performance, KU Leuven.
    8. Hafezali Iqbal Hussain & Janusz Grabara & Mohd Shahril Ahmad Razimi & Saeed Pahlevan Sharif, 2019. "Sustainability of Leverage Levels in Response to Shocks in Equity Prices: Islamic Finance as a Socially Responsible Investment," Sustainability, MDPI, Open Access Journal, vol. 11(12), pages 1-1, June.
    9. Konstantin Buechel & Martina Jakob & Daniel Steffen & Christoph Kuehnhanss & Aymo Brunetti, 2020. "The Relative Effectiveness of Teachers and Learning Software: Evidence from a Field Experiment in El Salvador," Diskussionsschriften dp2006, Universitaet Bern, Departement Volkswirtschaft.
    10. Andrés Ham & Darío Maldonado & Michael Weintraub & Andrés Felipe Camacho & Daniela Gualtero, 2019. "Reducing Alcohol-Related Violence: A Field Experiment with Bartenders," Documentos de trabajo 017834, Escuela de Gobierno - Universidad de los Andes.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bruno Ferman, 2019. "A simple way to assess inference methods," Papers 1912.08772,, revised Dec 2020.
    2. Knutsson, Daniel, 2020. "The Effect of Water Filtration on Cholera Mortality," Working Paper Series 1346, Research Institute of Industrial Economics.
    3. James G. MacKinnon & Matthew D. Webb, 2017. "Pitfalls When Estimating Treatment Effects Using Clustered Data," Working Paper 1387, Economics Department, Queen's University.
    4. Yichong Zhang & Xin Zheng, 2020. "Quantile treatment effects and bootstrap inference under covariate‐adaptive randomization," Quantitative Economics, Econometric Society, vol. 11(3), pages 957-982, July.
    5. Chor, Elise & Andresen, Martin Eckhoff & Kalil, Ariel, 2016. "The impact of universal prekindergarten on family behavior and child outcomes," Economics of Education Review, Elsevier, vol. 55(C), pages 168-181.
    6. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    7. Young, Alwyn, 2019. "Channeling Fisher: randomization tests and the statistical insignificance of seemingly significant experimental results," LSE Research Online Documents on Economics 101401, London School of Economics and Political Science, LSE Library.
    8. Heiko T. Burret & Lars P. Feld, 2018. "Vertical effects of fiscal rules: the Swiss experience," International Tax and Public Finance, Springer;International Institute of Public Finance, vol. 25(3), pages 673-721, June.
    9. MacKinnon, James G. & Webb, Matthew D., 2020. "Randomization inference for difference-in-differences with few treated clusters," Journal of Econometrics, Elsevier, vol. 218(2), pages 435-450.
    10. Bruno Ferman & Cristine Pinto, 2019. "Inference in Differences-in-Differences with Few Treated Groups and Heteroskedasticity," The Review of Economics and Statistics, MIT Press, vol. 101(3), pages 452-467, July.
    11. Clément de Chaisemartin & Jaime Ramirez-Cuellar, 2020. "At What Level Should One Cluster Standard Errors in Paired Experiments, and in Stratified Experiments with Small Strata?," NBER Working Papers 27609, National Bureau of Economic Research, Inc.
    12. Michael Pollmann, 2020. "Causal Inference for Spatial Treatments," Papers 2011.00373,
    13. Alexandre Belloni & Victor Chernozhukov & Denis Chetverikov & Christian Hansen & Kengo Kato, 2018. "High-dimensional econometrics and regularized GMM," CeMMAP working papers CWP35/18, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    14. Hollingsworth, Bruce & Ohinata, Asako & Picchio, Matteo & Walker, Ian, 2017. "Labour supply and informal care supply: The impacts of financial support for long-term elderly care," GLO Discussion Paper Series 118, Global Labor Organization (GLO).
    15. Congdon Fors, Heather & Houngbedji, Kenneth & Lindskog, Annika, 2019. "Land certification and schooling in rural Ethiopia," World Development, Elsevier, vol. 115(C), pages 190-208.
    16. Andrés Elberg & Pedro M. Gardete & Rosario Macera & Carlos Noton, 2019. "Dynamic effects of price promotions: field evidence, consumer search, and supply-side implications," Quantitative Marketing and Economics (QME), Springer, vol. 17(1), pages 1-58, March.
    17. Suresh de Mel & David McKenzie & Christopher Woodruff, 2019. "Labor Drops: Experimental Evidence on the Return to Additional Labor in Microenterprises," American Economic Journal: Applied Economics, American Economic Association, vol. 11(1), pages 202-235, January.
    18. Pedro Carneiro & Sokbae (Simon) Lee & Daniel Wilhelm, 2016. "Optimal data collection for randomized control trials," CeMMAP working papers CWP15/16, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    19. Ivan A Canay & Vishal Kamat, 2018. "Approximate Permutation Tests and Induced Order Statistics in the Regression Discontinuity Design," Review of Economic Studies, Oxford University Press, vol. 85(3), pages 1577-1608.
    20. James G. MacKinnon & Matthew D. Webb & Morten Ø. Nielsen, 2017. "Bootstrap And Asymptotic Inference With Multiway Clustering," Working Paper 1386, Economics Department, Queen's University.


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:tsj:stataj:y:17:y:2017:i:3:p:630-651. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Christopher F. Baum) or (Lisa Gilmore). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.