IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v47y2022i1p69-100.html
   My bibliography  Save this article

Block What You Can, Except When You Shouldn’t

Author

Listed:
  • Nicole E. Pashley

    (Rutgers University)

  • Luke W. Miratrix

    (1812Harvard University)

Abstract

Several branches of the potential outcome causal inference literature have discussed the merits of blocking versus complete randomization. Some have concluded it can never hurt the precision of estimates, and some have concluded it can hurt. In this article, we reconcile these apparently conflicting views, give a more thorough discussion of what guarantees no harm, and discuss how other aspects of a blocked design can cost, all in terms of estimator precision. We discuss how the different findings are due to different sampling models and assumptions of how the blocks were formed. We also connect these ideas to common misconceptions; for instance, we show that analyzing a blocked experiment as if it were completely randomized, a seemingly conservative method, can actually backfire in some cases. Overall, we find that blocking can have a price but that this price is usually small and the potential for gain can be large. It is hard to go too far wrong with blocking.

Suggested Citation

  • Nicole E. Pashley & Luke W. Miratrix, 2022. "Block What You Can, Except When You Shouldn’t," Journal of Educational and Behavioral Statistics, , vol. 47(1), pages 69-100, February.
    • Handle: RePEc:sae:jedbes:v:47:y:2022:i:1:p:69-100
    DOI: 10.3102/10769986211027240
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/10769986211027240
    Download Restriction: no
    File URL: https://libkey.io/10.3102/10769986211027240?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Kari Lock Morgan & Donald B. Rubin, 2015. "Rerandomization to Balance Tiers of Covariates," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(512), pages 1412-1421, December.
    2. Miratrix, Luke W. & Sekhon, Jasjeet S. & Theodoridis, Alexander G. & Campos, Luis F., 2018. "Worth Weighting? How to Think About and Use Weights in Survey Experiments," Political Analysis, Cambridge University Press, vol. 26(3), pages 275-291, July.
    3. Luke W. Miratrix & Jasjeet S. Sekhon & Bin Yu, 2013. "Adjusting treatment effect estimates by post-stratification in randomized experiments," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 75(2), pages 369-396, March.
    4. Nicole E. Pashley & Luke W. Miratrix, 2021. "Insights on Variance Estimation for Blocked and Matched Pairs Designs," Journal of Educational and Behavioral Statistics, , vol. 46(3), pages 271-296, June.
    5. Kosuke Imai & Gary King & Elizabeth A. Stuart, 2008. "Misunderstandings between experimentalists and observationalists about causal inference," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 171(2), pages 481-502, April.
    6. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nicole E. Pashley & Luke W. Miratrix, 2021. "Insights on Variance Estimation for Blocked and Matched Pairs Designs," Journal of Educational and Behavioral Statistics, , vol. 46(3), pages 271-296, June.
    2. Haoge Chang, 2023. "Design-based Estimation Theory for Complex Experiments," Papers 2311.06891, arXiv.org.
    3. Naoki Egami & Erin Hartman, 2021. "Covariate selection for generalizing experimental results: Application to a large‐scale development program in Uganda," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(4), pages 1524-1548, October.
    4. Pedro Carneiro & Sokbae Lee & Daniel Wilhelm, 2020. "Optimal data collection for randomized control trials [Microcredit impacts: Evidence from a randomized microcredit program placement experiment by Compartamos Banco]," The Econometrics Journal, Royal Economic Society, vol. 23(1), pages 1-31.
    5. Haoge Chang & Joel Middleton & P. M. Aronow, 2021. "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials," Papers 2110.08425, arXiv.org, revised Oct 2021.
    6. Adam Kapelner & Abba Krieger, 2023. "A matching procedure for sequential experiments that iteratively learns which covariates improve power," Biometrics, The International Biometric Society, vol. 79(1), pages 216-229, March.
    7. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," SciencePo Working papers Main hal-03455978, HAL.
    8. Tingting Zhou & Michael R. Elliott & Roderick J. A. Little, 2022. "Addressing Disparities in the Propensity Score Distributions for Treatment Comparisons from Observational Studies," Stats, MDPI, vol. 5(4), pages 1-17, December.
    9. Jan Stede, 2019. "Do Energy Efficiency Networks Save Energy? Evidence from German Plant-Level Data," Discussion Papers of DIW Berlin 1813, DIW Berlin, German Institute for Economic Research.
    10. Donald P. Green & Winston Lin & Claudia Gerber, 2018. "Optimal Allocation of Interviews to Baseline and Endline Surveys in Place-Based Randomized Trials and Quasi-Experiments," Evaluation Review, , vol. 42(4), pages 391-422, August.
    11. Marco Morucci & Md. Noor-E-Alam & Cynthia Rudin, 2022. "A Robust Approach to Quantifying Uncertainty in Matching Problems of Causal Inference," INFORMS Joural on Data Science, INFORMS, vol. 1(2), pages 156-171, October.
    12. Pedro Carneiro & Sokbae (Simon) Lee & Daniel Wilhelm, 2016. "Optimal data collection for randomized control trials," CeMMAP working papers 15/16, Institute for Fiscal Studies.
    13. Ding, Peng, 2021. "The Frisch–Waugh–Lovell theorem for standard errors," Statistics & Probability Letters, Elsevier, vol. 168(C).
    14. Peter Z. Schochet, 2020. "Analyzing Grouped Administrative Data for RCTs Using Design-Based Methods," Journal of Educational and Behavioral Statistics, , vol. 45(1), pages 32-57, February.
    15. Tenglong Li & Kenneth A. Frank & Mingming Chen, 2024. "A Conceptual Framework for Quantifying the Robustness of a Regression-Based Causal Inference in Observational Study," Mathematics, MDPI, vol. 12(3), pages 1-14, January.
    16. Takasaki, Yoshito, 2020. "Impacts of disability on poverty: Quasi-experimental evidence from landmine amputees in Cambodia," Journal of Economic Behavior & Organization, Elsevier, vol. 180(C), pages 85-107.
    17. Yuehao Bai, 2022. "Optimality of Matched-Pair Designs in Randomized Controlled Trials," Papers 2206.07845, arXiv.org.
    18. Esterling, Kevin M. & Brady, David & Schwitzgebel, Eric, 2023. "The Necessity of Construct and External Validity for Generalized Causal Claims," I4R Discussion Paper Series 18, The Institute for Replication (I4R).
    19. Ke Zhu & Hanzhong Liu, 2023. "Pair‐switching rerandomization," Biometrics, The International Biometric Society, vol. 79(3), pages 2127-2142, September.
    20. Yves Tillé, 2022. "Some Solutions Inspired by Survey Sampling Theory to Build Effective Clinical Trials," International Statistical Review, International Statistical Institute, vol. 90(3), pages 481-498, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:47:y:2022:i:1:p:69-100. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.