IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2405.03910.html
   My bibliography  Save this paper

A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances

Author

Listed:
  • Yuehao Bai
  • Azeem M. Shaikh
  • Max Tabord-Meehan

Abstract

The past two decades have witnessed a surge of new research in the analysis of randomized experiments. The emergence of this literature may seem surprising given the widespread use and long history of experiments as the "gold standard" in program evaluation, but this body of work has revealed many subtle aspects of randomized experiments that may have been previously unappreciated. This article provides an overview of some of these topics, primarily focused on stratification, regression adjustment, and cluster randomization.

Suggested Citation

  • Yuehao Bai & Azeem M. Shaikh & Max Tabord-Meehan, 2024. "A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances," Papers 2405.03910, arXiv.org.
    • Handle: RePEc:arx:papers:2405.03910
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2405.03910
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Kock, Anders Bredahl & Preinerstorfer, David & Veliyev, Bezirgen, 2023. "Treatment recommendation with distributional targets," Journal of Econometrics, Elsevier, vol. 234(2), pages 624-646.
    2. Jinyong Hahn & Keisuke Hirano & Dean Karlan, 2011. "Adaptive Experimental Design Using the Propensity Score," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(1), pages 96-108, January.
    3. Bingkai Wang & Ryoko Susukida & Ramin Mojtabai & Masoumeh Amin-Esmaeili & Michael Rosenblum, 2023. "Model-Robust Inference for Clinical Trials that Improve Precision by Stratified Randomization and Covariate Adjustment," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 118(542), pages 1152-1163, April.
    4. Miriam Bruhn & David McKenzie, 2009. "In Pursuit of Balance: Randomization in Practice in Development Field Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 200-232, October.
    5. John A. List & Azeem M. Shaikh & Yang Xu, 2019. "Multiple hypothesis testing in experimental economics," Experimental Economics, Springer;Economic Science Association, vol. 22(4), pages 773-793, December.
    6. Manski, Charles F, 1990. "Nonparametric Bounds on Treatment Effects," American Economic Review, American Economic Association, vol. 80(2), pages 319-323, May.
    7. Charles F. Manski, 2013. "Identification of treatment response with social interactions," Econometrics Journal, Royal Economic Society, vol. 16(1), pages 1-23, February.
    8. Toru Kitagawa & Aleksey Tetenov, 2021. "Equality-Minded Treatment Choice," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(2), pages 561-574, March.
    9. Romano, Joseph P. & Shaikh, Azeem M. & Wolf, Michael, 2008. "Formalized Data Snooping Based On Generalized Error Rates," Econometric Theory, Cambridge University Press, vol. 24(2), pages 404-447, April.
    10. Xin Lu & Tianle Liu & Hanzhong Liu & Peng Ding, 2023. "Design-based theory for cluster rerandomization," Biometrika, Biometrika Trust, vol. 110(2), pages 467-483.
    11. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2018. "Inference Under Covariate-Adaptive Randomization," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(524), pages 1784-1796, October.
    12. Keisuke Hirano & Jack R. Porter, 2009. "Asymptotics for Statistical Treatment Rules," Econometrica, Econometric Society, vol. 77(5), pages 1683-1701, September.
    13. Michael P. Leung, 2022. "Causal Inference Under Approximate Neighborhood Interference," Econometrica, Econometric Society, vol. 90(1), pages 267-293, January.
    14. John A. List & Azeem M. Shaikh & Atom Vayalinkal, 2023. "Multiple testing with covariate adjustment in experimental economics," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(6), pages 920-939, September.
    15. Dehejia, Rajeev H., 2005. "Program evaluation as a decision problem," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 141-173.
    16. Susan Athey & Dean Eckles & Guido W. Imbens, 2018. "Exact p-Values for Network Interference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(521), pages 230-240, January.
    17. Eric Mbakop & Max Tabord‐Meehan, 2021. "Model Selection for Treatment Choice: Penalized Welfare Maximization," Econometrica, Econometric Society, vol. 89(2), pages 825-848, March.
    18. Yuehao Bai & Hongchang Guo & Azeem M. Shaikh & Max Tabord-Meehan, 2023. "Inference in Experiments with Matched Pairs and Imperfect Compliance," Papers 2307.13094, arXiv.org, revised Jun 2024.
    19. Clément de Chaisemartin & Jaime Ramirez-Cuellar, 2024. "At What Level Should One Cluster Standard Errors in Paired and Small-Strata Experiments?," American Economic Journal: Applied Economics, American Economic Association, vol. 16(1), pages 193-212, January.
    20. Jiang, Liang & Phillips, Peter C.B. & Tao, Yubo & Zhang, Yichong, 2023. "Regression-adjusted estimation of quantile treatment effects under covariate-adaptive randomizations," Journal of Econometrics, Elsevier, vol. 234(2), pages 758-776.
    21. Yuehao Bai, 2022. "Optimality of Matched-Pair Designs in Randomized Controlled Trials," Papers 2206.07845, arXiv.org.
    22. Maximilian Kasy & Anja Sautmann, 2021. "Adaptive Treatment Assignment in Experiments for Policy Choice," Econometrica, Econometric Society, vol. 89(1), pages 113-132, January.
    23. Toru Kitagawa & Aleksey Tetenov, 2018. "Who Should Be Treated? Empirical Welfare Maximization Methods for Treatment Choice," Econometrica, Econometric Society, vol. 86(2), pages 591-616, March.
    24. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2019. "Inference under covariate‐adaptive randomization with multiple treatments," Quantitative Economics, Econometric Society, vol. 10(4), pages 1747-1785, November.
    25. Harold D Chiang & Yukitoshi Matsushita & Taisuke Otsu, 2023. "Regression adjustment in randomized controlled trials with many covariates," Papers 2302.00469, arXiv.org, revised Nov 2023.
    26. Rachel Glennerster & Kudzai Takavarasha, 2013. "Running Randomized Evaluations: A Practical Guide," Economics Books, Princeton University Press, edition 1, number 10085.
    27. Karthik Muralidharan & Paul Niehaus, 2017. "Experimentation at Scale," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 103-124, Fall.
    28. Yuehao Bai & Jizhou Liu & Azeem M. Shaikh & Max Tabord-Meehan, 2023. "On the Efficiency of Finely Stratified Experiments," Papers 2307.15181, arXiv.org, revised Aug 2024.
    29. Bai, Yuehao, 2023. "Why randomize? Minimax optimality under permutation invariance," Journal of Econometrics, Elsevier, vol. 232(2), pages 565-575.
    30. Yuchen Hu & Shuangning Li & Stefan Wager, 2022. "Average direct and indirect causal effects under interference [Estimating average causal effects under general interference, with application to a social network experiment]," Biometrika, Biometrika Trust, vol. 109(4), pages 1165-1172.
    31. Stefan Wager & Kuang Xu, 2021. "Experimenting in Equilibrium," Management Science, INFORMS, vol. 67(11), pages 6694-6715, November.
    32. Jiafeng Chen & Isaiah Andrews, 2023. "Optimal Conditional Inference in Adaptive Experiments," Papers 2309.12162, arXiv.org.
    33. Machado, Cecilia & Shaikh, Azeem M. & Vytlacil, Edward J., 2019. "Instrumental variables and the sign of the average treatment effect," Journal of Econometrics, Elsevier, vol. 212(2), pages 522-555.
    34. Bhattacharya, Debopam & Dupas, Pascaline, 2012. "Inferring welfare maximizing treatment assignment under budget constraints," Journal of Econometrics, Elsevier, vol. 167(1), pages 168-196.
    35. Heckman, James, 2013. "Sample selection bias as a specification error," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 31(3), pages 129-137.
    36. Yong Cai & Ahnaf Rafi, 2022. "On the Performance of the Neyman Allocation with Small Pilots," Papers 2206.04643, arXiv.org, revised Jun 2024.
    37. Federico A. Bugni & Mengsi Gao, 2021. "Inference under Covariate-Adaptive Randomization with Imperfect Compliance," Papers 2102.03937, arXiv.org, revised Jul 2023.
    38. Charles F. Manski, 2004. "Statistical Treatment Rules for Heterogeneous Populations," Econometrica, Econometric Society, vol. 72(4), pages 1221-1246, July.
    39. Soohyung Lee & Azeem M. Shaikh, 2014. "Multiple Testing And Heterogeneous Treatment Effects: Re‐Evaluating The Effect Of Progresa On School Enrollment," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(4), pages 612-626, June.
    40. Yuehao Bai, 2022. "Optimality of Matched-Pair Designs in Randomized Controlled Trials," American Economic Review, American Economic Association, vol. 112(12), pages 3911-3940, December.
    41. Guillaume Basse & Avi Feller, 2018. "Analyzing Two-Stage Experiments in the Presence of Interference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(521), pages 41-55, January.
    42. Harrison H. Li & Art B. Owen, 2023. "Double machine learning and design in batch adaptive experiments," Papers 2309.15297, arXiv.org.
    43. Zhao, Anqi & Ding, Peng, 2021. "Covariate-adjusted Fisher randomization tests for the average treatment effect," Journal of Econometrics, Elsevier, vol. 225(2), pages 278-294.
    44. Keisuke Hirano & Jack R. Porter, 2023. "Asymptotic Representations for Sequential Decisions, Adaptive Experiments, and Batched Bandits," Papers 2302.03117, arXiv.org.
    45. Akanksha Negi & Jeffrey M. Wooldridge, 2021. "Revisiting regression adjustment in experiments with heterogeneous treatment effects," Econometric Reviews, Taylor & Francis Journals, vol. 40(5), pages 504-534, April.
    46. Jason Wu & Peng Ding, 2021. "Randomization Tests for Weak Null Hypotheses in Randomized Experiments," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(536), pages 1898-1913, October.
    47. Davide Viviano & Lihua Lei & Guido Imbens & Brian Karrer & Okke Schrijvers & Liang Shi, 2023. "Causal clustering: design of cluster experiments under network interference," Papers 2310.14983, arXiv.org, revised Jan 2024.
    48. Vazquez-Bare, Gonzalo, 2023. "Identification and estimation of spillover effects in randomized experiments," Journal of Econometrics, Elsevier, vol. 237(1).
    49. Samuel Higbee, 2022. "Policy Learning with New Treatments," Papers 2210.04703, arXiv.org, revised Sep 2023.
    50. Fangzhou Su & Peng Ding, 2021. "Model‐assisted analyses of cluster‐randomized experiments," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(5), pages 994-1015, November.
    51. Joseph P. Romano & Michael Wolf, 2005. "Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 94-108, March.
    52. Jay Bhattacharya & Azeem M. Shaikh & Edward Vytlacil, 2008. "Treatment Effect Bounds under Monotonicity Assumptions: An Application to Swan-Ganz Catheterization," American Economic Review, American Economic Association, vol. 98(2), pages 351-356, May.
    53. Xinran Li & Peng Ding, 2017. "General Forms of Finite Population Central Limit Theorems with Applications to Causal Inference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(520), pages 1759-1769, October.
    54. Susan Athey & Stefan Wager, 2021. "Policy Learning With Observational Data," Econometrica, Econometric Society, vol. 89(1), pages 133-161, January.
    55. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769.
    56. Chan Park & Hyunseung Kang, 2023. "Assumption-Lean Analysis of Cluster Randomized Trials in Infectious Diseases for Intent-to-Treat Effects and Network Effects," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 118(542), pages 1195-1206, April.
    57. Laura Forastiere & Edoardo M. Airoldi & Fabrizia Mealli, 2021. "Identification and Estimation of Treatment and Interference Effects in Observational Studies on Networks," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(534), pages 901-918, April.
    58. Yuehao Bai & Joseph P. Romano & Azeem M. Shaikh, 2022. "Inference in Experiments With Matched Pairs," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 117(540), pages 1726-1737, October.
    59. Nicole E. Pashley & Luke W. Miratrix, 2021. "Insights on Variance Estimation for Blocked and Matched Pairs Designs," Journal of Educational and Behavioral Statistics, , vol. 46(3), pages 271-296, June.
    60. Bhattacharya, Jay & Shaikh, Azeem M. & Vytlacil, Edward, 2012. "Treatment effect bounds: An application to Swan–Ganz catheterization," Journal of Econometrics, Elsevier, vol. 168(2), pages 223-243.
    61. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    62. Max Cytrynbaum, 2021. "Optimal Stratification of Survey Experiments," Papers 2111.08157, arXiv.org, revised Aug 2023.
    63. Michael P. Leung, 2020. "Treatment and Spillover Effects Under Network Interference," The Review of Economics and Statistics, MIT Press, vol. 102(2), pages 368-380, May.
    64. G W Basse & A Feller & P Toulis, 2019. "Randomization tests of causal effects under interference," Biometrika, Biometrika Trust, vol. 106(2), pages 487-494.
    65. Kitagawa, Toru & Wang, Guanyi, 2023. "Who should get vaccinated? Individualized allocation of vaccines over SIR network," Journal of Econometrics, Elsevier, vol. 232(1), pages 109-131.
    66. Nathan Kallus, 2021. "On the optimality of randomization in experimental design: How to randomize for minimax variance and design‐based inference," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(2), pages 404-409, April.
    67. Harold D Chiang & Yukitoshi Matsushita & Taisuke Otsu, 2023. "Regression adjustment in randomized controlled trials with many covariates," STICERD - Econometrics Paper Series 627, Suntory and Toyota International Centres for Economics and Related Disciplines, LSE.
    68. Mengsi Gao & Peng Ding, 2023. "Causal inference in network experiments: regression-based analysis and design-based properties," Papers 2309.07476, arXiv.org, revised Nov 2023.
    69. Kosuke Imai & Zhichao Jiang & Anup Malani, 2021. "Causal Inference With Interference and Noncompliance in Two-Stage Randomized Experiments," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(534), pages 632-644, April.
    70. Hudgens, Michael G. & Halloran, M. Elizabeth, 2008. "Toward Causal Inference With Interference," Journal of the American Statistical Association, American Statistical Association, vol. 103, pages 832-842, June.
    71. Yuchen Hu & Shuangning Li & Stefan Wager, 2021. "Average Direct and Indirect Causal Effects under Interference," Papers 2104.03802, arXiv.org, revised Jan 2022.
    72. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    73. Eduardo M. Azevedo & Alex Deng & José Luis Montiel Olea & Justin Rao & E. Glen Weyl, 2020. "A/B Testing with Fat Tails," Journal of Political Economy, University of Chicago Press, vol. 128(12), pages 4614-4000.
    74. Zach Branson & inran Li & Peng Ding, 2024. "Power and sample size calculations for rerandomization," Biometrika, Biometrika Trust, vol. 111(1), pages 355-363.
    75. Yichong Zhang & Xin Zheng, 2020. "Quantile treatment effects and bootstrap inference under covariate‐adaptive randomization," Quantitative Economics, Econometric Society, vol. 11(3), pages 957-982, July.
    76. Ahnaf Rafi, 2023. "Efficient Semiparametric Estimation of Average Treatment Effects Under Covariate Adaptive Randomization," Papers 2305.08340, arXiv.org.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Fang Han, 2024. "An Introduction to Permutation Processes (version 0.5)," Papers 2407.09664, arXiv.org.
    2. Laurent Davezies & Guillaume Hollard & Pedro Vergara Merino, 2024. "Revisiting Randomization with the Cube Method," Papers 2407.13613, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Haoge Chang, 2023. "Design-based Estimation Theory for Complex Experiments," Papers 2311.06891, arXiv.org.
    2. Davide Viviano & Jess Rudder, 2020. "Policy design in experiments with unknown interference," Papers 2011.08174, arXiv.org, revised May 2024.
    3. Jinglong Zhao, 2024. "Experimental Design For Causal Inference Through An Optimization Lens," Papers 2408.09607, arXiv.org, revised Aug 2024.
    4. Kitagawa, Toru & Wang, Guanyi, 2023. "Who should get vaccinated? Individualized allocation of vaccines over SIR network," Journal of Econometrics, Elsevier, vol. 232(1), pages 109-131.
    5. Davide Viviano, 2019. "Policy Targeting under Network Interference," Papers 1906.10258, arXiv.org, revised Apr 2024.
    6. Toru Kitagawa & Guanyi Wang, 2021. "Who should get vaccinated? Individualized allocation of vaccines over SIR network," CeMMAP working papers CWP28/21, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    7. Susan Athey & Guido Imbens, 2016. "The Econometrics of Randomized Experiments," Papers 1607.00698, arXiv.org.
    8. Tadao Hoshino & Takahide Yanagi, 2021. "Causal Inference with Noncompliance and Unknown Interference," Papers 2108.07455, arXiv.org, revised Oct 2023.
    9. Yi Zhang & Kosuke Imai, 2023. "Individualized Policy Evaluation and Learning under Clustered Network Interference," Papers 2311.02467, arXiv.org, revised Feb 2024.
    10. Han, Kevin & Basse, Guillaume & Bojinov, Iavor, 2024. "Population interference in panel experiments," Journal of Econometrics, Elsevier, vol. 238(1).
    11. Jiang, Liang & Phillips, Peter C.B. & Tao, Yubo & Zhang, Yichong, 2023. "Regression-adjusted estimation of quantile treatment effects under covariate-adaptive randomizations," Journal of Econometrics, Elsevier, vol. 234(2), pages 758-776.
    12. Susan Athey & Stefan Wager, 2021. "Policy Learning With Observational Data," Econometrica, Econometric Society, vol. 89(1), pages 133-161, January.
    13. Zhao, Anqi & Ding, Peng, 2024. "No star is good news: A unified look at rerandomization based on p-values from covariate balance tests," Journal of Econometrics, Elsevier, vol. 241(1).
    14. Zhao, Anqi & Ding, Peng, 2021. "Covariate-adjusted Fisher randomization tests for the average treatment effect," Journal of Econometrics, Elsevier, vol. 225(2), pages 278-294.
    15. David M. Ritzwoller & Joseph P. Romano & Azeem M. Shaikh, 2024. "Randomization Inference: Theory and Applications," Papers 2406.09521, arXiv.org.
    16. Kock, Anders Bredahl & Preinerstorfer, David & Veliyev, Bezirgen, 2023. "Treatment recommendation with distributional targets," Journal of Econometrics, Elsevier, vol. 234(2), pages 624-646.
    17. Liang Jiang & Oliver B. Linton & Haihan Tang & Yichong Zhang, 2022. "Improving Estimation Efficiency via Regression-Adjustment in Covariate-Adaptive Randomizations with Imperfect Compliance," Papers 2201.13004, arXiv.org, revised Jun 2023.
    18. Anders Bredahl Kock & David Preinerstorfer, 2024. "Regularizing Discrimination in Optimal Policy Learning with Distributional Targets," Papers 2401.17909, arXiv.org.
    19. Yuehao Bai & Jizhou Liu & Azeem M. Shaikh & Max Tabord-Meehan, 2023. "On the Efficiency of Finely Stratified Experiments," Papers 2307.15181, arXiv.org, revised Aug 2024.
    20. Toru Kitagawa & Guanyi Wang, 2020. "Who should get vaccinated? Individualized allocation of vaccines over SIR network," CeMMAP working papers CWP59/20, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2405.03910. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.