IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v47y2022i1p69-100.html

Block What You Can, Except When You Shouldn’t

Author

Listed:
  • Nicole E. Pashley

    (Rutgers University)

  • Luke W. Miratrix

    (1812Harvard University)

Abstract

Several branches of the potential outcome causal inference literature have discussed the merits of blocking versus complete randomization. Some have concluded it can never hurt the precision of estimates, and some have concluded it can hurt. In this article, we reconcile these apparently conflicting views, give a more thorough discussion of what guarantees no harm, and discuss how other aspects of a blocked design can cost, all in terms of estimator precision. We discuss how the different findings are due to different sampling models and assumptions of how the blocks were formed. We also connect these ideas to common misconceptions; for instance, we show that analyzing a blocked experiment as if it were completely randomized, a seemingly conservative method, can actually backfire in some cases. Overall, we find that blocking can have a price but that this price is usually small and the potential for gain can be large. It is hard to go too far wrong with blocking.

Suggested Citation

  • Nicole E. Pashley & Luke W. Miratrix, 2022. "Block What You Can, Except When You Shouldn’t," Journal of Educational and Behavioral Statistics, , vol. 47(1), pages 69-100, February.
    • Handle: RePEc:sae:jedbes:v:47:y:2022:i:1:p:69-100
    DOI: 10.3102/10769986211027240
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/10769986211027240
    Download Restriction: no
    File URL: https://libkey.io/10.3102/10769986211027240?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Luke W. Miratrix & Jasjeet S. Sekhon & Bin Yu, 2013. "Adjusting treatment effect estimates by post-stratification in randomized experiments," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 75(2), pages 369-396, March.
    2. Nicole E. Pashley & Luke W. Miratrix, 2021. "Insights on Variance Estimation for Blocked and Matched Pairs Designs," Journal of Educational and Behavioral Statistics, , vol. 46(3), pages 271-296, June.
    3. Kari Lock Morgan & Donald B. Rubin, 2015. "Rerandomization to Balance Tiers of Covariates," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(512), pages 1412-1421, December.
    4. Kosuke Imai & Gary King & Elizabeth A. Stuart, 2008. "Misunderstandings between experimentalists and observationalists about causal inference," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 171(2), pages 481-502, April.
    5. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, Enero-Abr.
    6. Miratrix, Luke W. & Sekhon, Jasjeet S. & Theodoridis, Alexander G. & Campos, Luis F., 2018. "Worth Weighting? How to Think About and Use Weights in Survey Experiments," Political Analysis, Cambridge University Press, vol. 26(3), pages 275-291, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Xiao Liu, 2026. "Estimating Causal Mediation Effects in Multiple-Mediator Analyses With Clustered Data," Journal of Educational and Behavioral Statistics, , vol. 51(2), pages 310-343, April.
    2. Gregory Chernov, 2025. "The Alternative Factors Leading to Replication Crisis: Prediction and Evaluation," Evaluation Review, , vol. 49(1), pages 147-164, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhao, Anqi & Ding, Peng, 2024. "No star is good news: A unified look at rerandomization based on p-values from covariate balance tests," Journal of Econometrics, Elsevier, vol. 241(1).
    2. Nicole E. Pashley & Luke W. Miratrix, 2021. "Insights on Variance Estimation for Blocked and Matched Pairs Designs," Journal of Educational and Behavioral Statistics, , vol. 46(3), pages 271-296, June.
    3. Haoge Chang, 2023. "Design-based Estimation Theory for Complex Experiments," Papers 2311.06891, arXiv.org, revised May 2025.
    4. Naoki Egami & Erin Hartman, 2021. "Covariate selection for generalizing experimental results: Application to a large‐scale development program in Uganda," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(4), pages 1524-1548, October.
    5. Pedro Carneiro & Sokbae Lee & Daniel Wilhelm, 2020. "Optimal data collection for randomized control trials," The Econometrics Journal, Royal Economic Society, vol. 23(1), pages 1-31.
    6. Haoge Chang & Joel Middleton & P. M. Aronow, 2021. "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials," Papers 2110.08425, arXiv.org, revised Oct 2021.
    7. Adam Kapelner & Abba Krieger, 2023. "A matching procedure for sequential experiments that iteratively learns which covariates improve power," Biometrics, The International Biometric Society, vol. 79(1), pages 216-229, March.
    8. Tingting Zhou & Michael R. Elliott & Roderick J. A. Little, 2022. "Addressing Disparities in the Propensity Score Distributions for Treatment Comparisons from Observational Studies," Stats, MDPI, vol. 5(4), pages 1-17, December.
    9. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," Sciences Po Economics Publications (main) hal-03455978, HAL.
    10. Jan Stede, 2019. "Do Energy Efficiency Networks Save Energy? Evidence from German Plant-Level Data," Discussion Papers of DIW Berlin 1813, DIW Berlin, German Institute for Economic Research.
    11. Donald P. Green & Winston Lin & Claudia Gerber, 2018. "Optimal Allocation of Interviews to Baseline and Endline Surveys in Place-Based Randomized Trials and Quasi-Experiments," Evaluation Review, , vol. 42(4), pages 391-422, August.
    12. Marco Morucci & Md. Noor-E-Alam & Cynthia Rudin, 2022. "A Robust Approach to Quantifying Uncertainty in Matching Problems of Causal Inference," INFORMS Joural on Data Science, INFORMS, vol. 1(2), pages 156-171, October.
    13. Ding, Peng, 2021. "The Frisch–Waugh–Lovell theorem for standard errors," Statistics & Probability Letters, Elsevier, vol. 168(C).
    14. Peter Z. Schochet, 2020. "Analyzing Grouped Administrative Data for RCTs Using Design-Based Methods," Journal of Educational and Behavioral Statistics, , vol. 45(1), pages 32-57, February.
    15. Tenglong Li & Kenneth A. Frank & Mingming Chen, 2024. "A Conceptual Framework for Quantifying the Robustness of a Regression-Based Causal Inference in Observational Study," Mathematics, MDPI, vol. 12(3), pages 1-14, January.
    16. Takasaki, Yoshito, 2020. "Impacts of disability on poverty: Quasi-experimental evidence from landmine amputees in Cambodia," Journal of Economic Behavior & Organization, Elsevier, vol. 180(C), pages 85-107.
    17. Yuehao Bai, 2022. "Optimality of Matched-Pair Designs in Randomized Controlled Trials," Papers 2206.07845, arXiv.org.
    18. Esterling, Kevin M. & Brady, David & Schwitzgebel, Eric, 2023. "The Necessity of Construct and External Validity for Generalized Causal Claims," I4R Discussion Paper Series 18, The Institute for Replication (I4R).
    19. Aronow, P.M. & Chang, Haoge & Lopatto, Patrick, 2025. "Fast computation of exact confidence intervals for randomized experiments with binary outcomes," Journal of Econometrics, Elsevier, vol. 251(C).
    20. Ron Berman & Elea McDonnell Feit, 2024. "Latent Stratification for Incrementality Experiments," Marketing Science, INFORMS, vol. 43(4), pages 903-917, July.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:47:y:2022:i:1:p:69-100. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.