IDEAS home Printed from https://ideas.repec.org/a/tsj/stataj/y17y2017i3p630-651.html
   My bibliography  Save this article

Randomization inference with Stata: A guide and software

Author

Listed:
  • Simon Heß

    (Goethe Universität Frankfurt am Main)

Abstract

Randomization inference or permutation tests are only sporadically used in economics and other social sciences—this despite a steep increase in ran- domization in field and laboratory experiments that provide perfect experimental setups for applying randomization inference. In the context of causal inference, such tests can handle problems often faced by applied researchers, including issues arising in the context of small samples, stratified or clustered treatment assign- ments, or nonstandard randomization techniques. Standard statistical software packages have either no implementation of randomization tests or very basic im- plementations. Whenever researchers use randomization inference, they regularly code individual program routines, risking inconsistencies and coding mistakes. In this article, I show how randomization inference can best be conducted in Stata and introduce a new command, ritest, to simplify such analyses. I illustrate this approach’s usefulness by replicating the results in Fujiwara and Wantchekon (2013, American Economic Journal: Applied Economics 5: 241–255) and running simulations. The applications cover clustered and stratified assignments, with varying cluster sizes, pairwise randomization, and the computation of nonapprox- imate p-values. The applications also touch upon joint hypothesis testing with randomization inference.

Suggested Citation

  • Simon Heß, 2017. "Randomization inference with Stata: A guide and software," Stata Journal, StataCorp LP, vol. 17(3), pages 630-651, September.
  • Handle: RePEc:tsj:stataj:y:17:y:2017:i:3:p:630-651
    Note: to access software from within Stata, net describe http://www.stata-journal.com/software/sj17-3/st0489/
    as

    Download full text from publisher

    File URL: http://www.stata-journal.com/article.html?article=st0489
    File Function: link to article purchase
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. MacKinnon, James G. & Webb, Matthew D., 2020. "Randomization inference for difference-in-differences with few treated clusters," Journal of Econometrics, Elsevier, vol. 218(2), pages 435-450.
    2. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2018. "Inference Under Covariate-Adaptive Randomization," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(524), pages 1784-1796, October.
    3. Joseph P. Romano & Michael Wolf, 2005. "Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 94-108, March.
    4. A. Colin Cameron & Douglas L. Miller, 2015. "A Practitioner’s Guide to Cluster-Robust Inference," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 317-372.
    5. James G. MacKinnon & Matthew D. Webb, 2017. "Wild Bootstrap Inference for Wildly Different Cluster Sizes," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(2), pages 233-254, March.
    6. Miriam Bruhn & David McKenzie, 2009. "In Pursuit of Balance: Randomization in Practice in Development Field Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 200-232, October.
    7. Peter Ganong & Simon Jäger, 2018. "A Permutation Test for the Regression Kink Design," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(522), pages 494-504, April.
    8. Jessica Cohen & Pascaline Dupas, 2010. "Free Distribution or Cost-Sharing? Evidence from a Randomized Malaria Prevention Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 125(1), pages 1-45.
    9. Burt S. Barnow & Matias D. Cattaneo & Rocío Titiunik & Gonzalo Vazquez‐Bare, 2017. "Comparing Inference Approaches for RD Designs: A Reexamination of the Effect of Head Start on Child Mortality," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 36(3), pages 643-681, June.
    10. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, November.
    11. Edward E. Leamer, 2010. "Tantalus on the Road to Asymptopia," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 31-46, Spring.
    12. Thomas Fujiwara & Leonard Wantchekon, 2013. "Can Informed Public Deliberation Overcome Clientelism? Experimental Evidence from Benin," American Economic Journal: Applied Economics, American Economic Association, vol. 5(4), pages 241-255, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Matthew D. Webb & James MacKinnon & Morten Nielsen, 2021. "Cluster–robust inference: A guide to empirical practice," Economics Virtual Symposium 2021 6, Stata Users Group.
    2. Bruno Ferman, 2019. "Assessing Inference Methods," Papers 1912.08772, arXiv.org, revised Oct 2021.
    3. James MacKinnon & Morten Ørregaard Nielsen, 2022. "Cluster-Robust Inference: A Guide to Empirical Practice," CREATES Research Papers 2022-08, Department of Economics and Business Economics, Aarhus University.
    4. Zhao, Anqi & Ding, Peng, 2021. "Covariate-adjusted Fisher randomization tests for the average treatment effect," Journal of Econometrics, Elsevier, vol. 225(2), pages 278-294.
    5. Cl'ement de Chaisemartin & Jaime Ramirez-Cuellar, 2019. "At What Level Should One Cluster Standard Errors in Paired and Small-Strata Experiments?," Papers 1906.00288, arXiv.org, revised May 2021.
    6. Chor, Elise & Andresen, Martin Eckhoff & Kalil, Ariel, 2016. "The impact of universal prekindergarten on family behavior and child outcomes," Economics of Education Review, Elsevier, vol. 55(C), pages 168-181.
    7. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    8. Bruno Ferman & Cristine Pinto, 2019. "Inference in Differences-in-Differences with Few Treated Groups and Heteroskedasticity," The Review of Economics and Statistics, MIT Press, vol. 101(3), pages 452-467, July.
    9. Knutsson, Daniel, 2020. "The Effect of Water Filtration on Cholera Mortality," Working Paper Series 1346, Research Institute of Industrial Economics.
    10. James G. MacKinnon & Matthew D. Webb, 2017. "Pitfalls When Estimating Treatment Effects Using Clustered Data," Working Paper 1387, Economics Department, Queen's University.
    11. Yichong Zhang & Xin Zheng, 2020. "Quantile treatment effects and bootstrap inference under covariate‐adaptive randomization," Quantitative Economics, Econometric Society, vol. 11(3), pages 957-982, July.
    12. James G. MacKinnon & Morten Ørregaard Nielsen & Matthew D. Webb, 2022. "Leverage, Influence, and the Jackknife in Clustered Regression Models: Reliable Inference Using summclust," Working Paper 1483, Economics Department, Queen's University.
    13. Young, Alwyn, 2019. "Channeling Fisher: randomization tests and the statistical insignificance of seemingly significant experimental results," LSE Research Online Documents on Economics 101401, London School of Economics and Political Science, LSE Library.
    14. Heiko T. Burret & Lars P. Feld, 2018. "Vertical effects of fiscal rules: the Swiss experience," International Tax and Public Finance, Springer;International Institute of Public Finance, vol. 25(3), pages 673-721, June.
    15. MacKinnon, James G. & Webb, Matthew D., 2020. "Randomization inference for difference-in-differences with few treated clusters," Journal of Econometrics, Elsevier, vol. 218(2), pages 435-450.
    16. Jeffrey D. Michler & Anna Josephson, 2021. "Recent Developments in Inference: Practicalities for Applied Economics," Papers 2107.09736, arXiv.org.
    17. Michael Pollmann, 2020. "Causal Inference for Spatial Treatments," Papers 2011.00373, arXiv.org.
    18. Alexandre Belloni & Victor Chernozhukov & Denis Chetverikov & Christian Hansen & Kengo Kato, 2018. "High-dimensional econometrics and regularized GMM," CeMMAP working papers CWP35/18, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    19. Hollingsworth, Bruce & Ohinata, Asako & Picchio, Matteo & Walker, Ian, 2017. "Labour supply and informal care supply: The impacts of financial support for long-term elderly care," GLO Discussion Paper Series 118, Global Labor Organization (GLO).
    20. Andrés Elberg & Pedro M. Gardete & Rosario Macera & Carlos Noton, 2019. "Dynamic effects of price promotions: field evidence, consumer search, and supply-side implications," Quantitative Marketing and Economics (QME), Springer, vol. 17(1), pages 1-58, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:tsj:stataj:y:17:y:2017:i:3:p:630-651. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: http://www.stata-journal.com/ .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Christopher F. Baum or Lisa Gilmore (email available below). General contact details of provider: http://www.stata-journal.com/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.