IDEAS home Printed from https://ideas.repec.org/a/eee/ecolet/v186y2020ics0165176519303878.html

Balancing tests in stratified randomized controlled trials: A cautionary note

Author

Listed:
  • Firpo, Sergio
  • Foguel, Miguel N.
  • Jales, Hugo

Abstract

We study the properties of imbalances tests of predetermined characteristics (covariates) in stratified randomized controlled trials. When the parameter of interest is a weighted average of the causal effects at all strata, then a test statistic based on a fixed effects regression model may suffice to detect the relevant imbalances. This will be true under a condition on the partial correlation between covariates and the outcome variable (uniformity condition). If that condition is violated, then the test based on the fixed effects regression will lack power. In that case, a test based on the fully saturated regression model can overcome this problem. Finally, we provide an omnibus test for the relevance of controlling for the covariates when estimating the average treatment effect (ATE).

Suggested Citation

  • Firpo, Sergio & Foguel, Miguel N. & Jales, Hugo, 2020. "Balancing tests in stratified randomized controlled trials: A cautionary note," Economics Letters, Elsevier, vol. 186(C).
    • Handle: RePEc:eee:ecolet:v:186:y:2020:i:c:s0165176519303878
    DOI: 10.1016/j.econlet.2019.108771
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0165176519303878
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.econlet.2019.108771?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. repec:hal:pseose:halshs-00840901 is not listed on IDEAS
    2. Bruno Crépon & Esther Duflo & Marc Gurgand & Roland Rathelot & Philippe Zamora, 2013. "Do Labor Market Policies have Displacement Effects? Evidence from a Clustered Randomized Experiment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 128(2), pages 531-580.
    3. Jerry Hausman, 2015. "Specification tests in econometrics," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 38(2), pages 112-134.
    4. Deaton, Angus & Cartwright, Nancy, 2018. "Understanding and misunderstanding randomized controlled trials," Social Science & Medicine, Elsevier, vol. 210(C), pages 2-21.
    5. Alexandre Belloni & Victor Chernozhukov & Christian Hansen, 2014. "Inference on Treatment Effects after Selection among High-Dimensional Controlsâ€," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 81(2), pages 608-650.
    6. Victor Chernozhukov & Denis Chetverikov & Mert Demirer & Esther Duflo & Christian Hansen & Whitney Newey & James Robins, 2018. "Double/debiased machine learning for treatment and structural parameters," Econometrics Journal, Royal Economic Society, vol. 21(1), pages 1-68, February.
    7. Krueger, Alan B & Whitmore, Diane M, 2001. "The Effect of Attending a Small Class in the Early Grades on College-Test Taking and Middle School Test Results: Evidence from Project STAR," Economic Journal, Royal Economic Society, vol. 111(468), pages 1-28, January.
    8. Miriam Bruhn & David McKenzie, 2009. "In Pursuit of Balance: Randomization in Practice in Development Field Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 200-232, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Piza, Caio & Zwager, Astrid & Ruzzante, Matteo & Dantas, Rafael & Loureiro, Andre, 2024. "Teacher-led innovations to improve education outcomes: Experimental evidence from Brazil," Journal of Public Economics, Elsevier, vol. 234(C).
    2. Brandon Genetin & Joyce Chen & Vladimir Kogan & Alan Kalish, 2022. "Mitigating implicit bias in student evaluations: A randomized intervention," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 44(1), pages 110-128, March.
    3. Wanda Schleder, 2025. "Effects of a Factorial Intervention on Emotional Skills," Working Papers 2507, Gutenberg School of Management and Economics, Johannes Gutenberg-Universität Mainz.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    2. Kyle Colangelo & Ying-Ying Lee, 2019. "Double debiased machine learning nonparametric inference with continuous treatments," CeMMAP working papers CWP72/19, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    3. Khanh Duong, 2024. "Is meritocracy just? New evidence from Boolean analysis and Machine learning," Journal of Computational Social Science, Springer, vol. 7(2), pages 1795-1821, October.
    4. Omar Al-Ubaydli & John List & Claire Mackevicius & Min Sok Lee & Dana Suskind, 2019. "How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling," Artefactual Field Experiments 00679, The Field Experiments Website.
    5. Guo, Jiaqi & Wang, Qiang & Li, Rongrong, 2024. "Can official development assistance promote renewable energy in sub-Saharan Africa countries? A matter of institutional transparency of recipient countries," Energy Policy, Elsevier, vol. 186(C).
    6. Bah, Tijan L. & Batista, Catia & Gubert, Flore & McKenzie, David, 2023. "Can information and alternatives to irregular migration reduce “backway” migration from The Gambia?," Journal of Development Economics, Elsevier, vol. 165(C).
    7. Tim Munday & James Brookes, 2021. "Mark my words: the transmission of central bank communication to the general public via the print media," Bank of England working papers 944, Bank of England.
    8. Girum Abebe & A Stefano Caria & Marcel Fafchamps & Paolo Falco & Simon Franklin & Simon Quinn, 2021. "Anonymity or Distance? Job Search and Labour Market Exclusion in a Growing African City [Endogenous Stratification in Randomized Experiments]," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 88(3), pages 1279-1310.
    9. Novella, Rafael & Rosas-Shady, David & Freund, Richard, 2024. "Is online job training for all? Experimental evidence on the effects of a Coursera program in Costa Rica," Journal of Development Economics, Elsevier, vol. 169(C).
    10. Kyle Colangelo & Ying-Ying Lee, 2019. "Double debiased machine learning nonparametric inference with continuous treatments," CeMMAP working papers CWP54/19, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    11. Heinisch, Katja & Scaramella, Fabio & Schult, Christoph, 2025. "Assumption errors and forecast accuracy: A partial linear instrumental variable and double machine learning approach," IWH Discussion Papers 6/2025, Halle Institute for Economic Research (IWH).
    12. Wang, Yuhao & Li, Xinran, 2025. "Asymptotic theory of the best-choice rerandomization using the Mahalanobis distance," Journal of Econometrics, Elsevier, vol. 251(C).
    13. Girum Abebe & Stefano Caria & Marcel Fafchamps & Paolo Falco & Simon Franklin & Simon Quinn, 2016. "Curse of Anonymity or Tyranny of Distance? The Impacts of Job-Search Support in Urban Ethiopia," CSAE Working Paper Series 2016-10, Centre for the Study of African Economies, University of Oxford.
    14. Roland G. Fryer, Jr, 2010. "Financial Incentives and Student Achievement: Evidence from Randomized Trials," NBER Working Papers 15898, National Bureau of Economic Research, Inc.
    15. Krikamol Muandet & Wittawat Jitkrittum & Jonas Kubler, 2020. "Kernel Conditional Moment Test via Maximum Moment Restriction," Papers 2002.09225, arXiv.org, revised Jun 2020.
    16. Abebe, Girum & Caria, Stefano & Fafchamps, Marcel & Falco, Paolo & Franklin, Simon & Quinn, Simon & Shilpi, Forhad, 2017. "Matching firms and workers in a field experiment in Ethiopia," LSE Research Online Documents on Economics 86572, London School of Economics and Political Science, LSE Library.
    17. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," The World Bank Research Observer, World Bank, vol. 33(1), pages 34-64.
    18. Martin Huber & Sarina Joy Oberhansli, 2026. "Difference-in-differences for mediation analysis using double machine learning," Papers 2602.23877, arXiv.org.
    19. Philipp Bach & Victor Chernozhukov & Malte S. Kurz & Martin Spindler & Sven Klaassen, 2021. "DoubleML -- An Object-Oriented Implementation of Double Machine Learning in R," Papers 2103.09603, arXiv.org, revised Jun 2024.
    20. Antonelli Joseph & Cefalu Matthew, 2020. "Averaging causal estimators in high dimensions," Journal of Causal Inference, De Gruyter, vol. 8(1), pages 92-107, January.

    More about this item

    Keywords

    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ecolet:v:186:y:2020:i:c:s0165176519303878. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ecolet .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.