IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v46y2021i3p271-296.html
   My bibliography  Save this article

Insights on Variance Estimation for Blocked and Matched Pairs Designs

Author

Listed:
  • Nicole E. Pashley
  • Luke W. Miratrix

    (1812Harvard University)

Abstract

Evaluating blocked randomized experiments from a potential outcomes perspective has two primary branches of work. The first focuses on larger blocks, with multiple treatment and control units in each block. The second focuses on matched pairs, with a single treatment and control unit in each block. These literatures not only provide different estimators for the standard errors of the estimated average impact, but they are also built on different sets of assumptions. Neither literature handles cases with blocks of varying size that contain singleton treatment or control units, a case which can occur in a variety of contexts, such as with different forms of matching or poststratification. In this article, we reconcile the literatures by carefully examining the performance of variance estimators under several different frameworks. We then use these insights to derive novel variance estimators for experiments containing blocks of different sizes.

Suggested Citation

  • Nicole E. Pashley & Luke W. Miratrix, 2021. "Insights on Variance Estimation for Blocked and Matched Pairs Designs," Journal of Educational and Behavioral Statistics, , vol. 46(3), pages 271-296, June.
    • Handle: RePEc:sae:jedbes:v:46:y:2021:i:3:p:271-296
    DOI: 10.3102/1076998620946272
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998620946272
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998620946272?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Robert J. LaLonde, 1984. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," Working Papers 563, Princeton University, Department of Economics, Industrial Relations Section..
    2. repec:adr:anecst:y:2008:i:91-92:p:09 is not listed on IDEAS
    3. Alberto Abadie & Guido W. Imbens, 2008. "Estimation of the Conditional Variance in Paired Experiments," Annals of Economics and Statistics, GENES, issue 91-92, pages 175-187.
    4. Miratrix, Luke W. & Sekhon, Jasjeet S. & Theodoridis, Alexander G. & Campos, Luis F., 2018. "Worth Weighting? How to Think About and Use Weights in Survey Experiments," Political Analysis, Cambridge University Press, vol. 26(3), pages 275-291, July.
    5. Rahul Mukerjee & Tirthankar Dasgupta & Donald B. Rubin, 2018. "Using Standard Tools From Finite Population Sampling to Improve Causal Inference for Complex Experiments," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(522), pages 868-881, April.
    6. Luke W. Miratrix & Jasjeet S. Sekhon & Bin Yu, 2013. "Adjusting treatment effect estimates by post-stratification in randomized experiments," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 75(2), pages 369-396, March.
    7. Ben B. Hansen, 2004. "Full Matching in an Observational Study of Coaching for the SAT," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 609-618, January.
    8. Qingyuan Zhao & Dylan S. Small & Paul R. Rosenbaum, 2018. "Cross-Screening in Observational Studies That Test Many Hypotheses," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(523), pages 1070-1084, July.
    9. Colin B. Fogarty, 2018. "On mitigating the analytical limitations of finely stratified experiments," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 80(5), pages 1035-1056, November.
    10. Kosuke Imai & Gary King & Elizabeth A. Stuart, 2008. "Misunderstandings between experimentalists and observationalists about causal inference," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 171(2), pages 481-502, April.
    11. Iacus, Stefano M. & King, Gary & Porro, Giuseppe, 2012. "Causal Inference without Balance Checking: Coarsened Exact Matching," Political Analysis, Cambridge University Press, vol. 20(1), pages 1-24, January.
    12. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, December.
    13. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-620, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Colin B. Fogarty, 2023. "Testing weak nulls in matched observational studies," Biometrics, The International Biometric Society, vol. 79(3), pages 2196-2207, September.
    2. Nicole E. Pashley & Luke W. Miratrix, 2022. "Block What You Can, Except When You Shouldn’t," Journal of Educational and Behavioral Statistics, , vol. 47(1), pages 69-100, February.
    3. Dmitry Arkhangelsky & Guido W. Imbens & Lihua Lei & Xiaoman Luo, 2021. "Design-Robust Two-Way-Fixed-Effects Regression For Panel Data," Papers 2107.13737, arXiv.org, revised Mar 2024.
    4. Yuehao Bai & Azeem M. Shaikh & Max Tabord-Meehan, 2024. "A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances," Papers 2405.03910, arXiv.org, revised Apr 2025.
    5. Yuehao Bai & Xun Huang & Joseph P. Romano & Azeem M. Shaikh & Max Tabord-Meehan, 2025. "A New Design-Based Variance Estimator for Finely Stratified Experiments," Papers 2503.10851, arXiv.org, revised Apr 2025.
    6. Zhao, Anqi & Ding, Peng, 2024. "No star is good news: A unified look at rerandomization based on p-values from covariate balance tests," Journal of Econometrics, Elsevier, vol. 241(1).
    7. Haoge Chang, 2023. "Design-based Estimation Theory for Complex Experiments," Papers 2311.06891, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," SciencePo Working papers Main hal-03455978, HAL.
    2. Jason J. Sauppe & Sheldon H. Jacobson & Edward C. Sewell, 2014. "Complexity and Approximation Results for the Balance Optimization Subset Selection Model for Causal Inference in Observational Studies," INFORMS Journal on Computing, INFORMS, vol. 26(3), pages 547-566, August.
    3. Gary King & Christopher Lucas & Richard A. Nielsen, 2017. "The Balance‐Sample Size Frontier in Matching Methods for Causal Inference," American Journal of Political Science, John Wiley & Sons, vol. 61(2), pages 473-489, April.
    4. Jason J. Sauppe & Sheldon H. Jacobson, 2017. "The role of covariate balance in observational studies," Naval Research Logistics (NRL), John Wiley & Sons, vol. 64(4), pages 323-344, June.
    5. Adeola Oyenubi & Martin Wittenberg, 2021. "Does the choice of balance-measure matter under genetic matching?," Empirical Economics, Springer, vol. 61(1), pages 489-502, July.
    6. Nicole E. Pashley & Luke W. Miratrix, 2022. "Block What You Can, Except When You Shouldn’t," Journal of Educational and Behavioral Statistics, , vol. 47(1), pages 69-100, February.
    7. Sergio Afcha & Jose García-Quevedo, 2016. "The impact of R&D subsidies on R&D employment composition," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 25(6), pages 955-975.
    8. Susan Athey & Guido W. Imbens & Stefan Wager, 2018. "Approximate residual balancing: debiased inference of average treatment effects in high dimensions," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 80(4), pages 597-623, September.
    9. Tymon Słoczyński, 2022. "Interpreting OLS Estimands When Treatment Effects Are Heterogeneous: Smaller Groups Get Larger Weights," The Review of Economics and Statistics, MIT Press, vol. 104(3), pages 501-509, May.
    10. Sloczynski, Tymon, 2018. "A General Weighted Average Representation of the Ordinary and Two-Stage Least Squares Estimands," IZA Discussion Papers 11866, Institute of Labor Economics (IZA).
    11. Katherine Baicker & Theodore Svoronos, 2019. "Testing the Validity of the Single Interrupted Time Series Design," NBER Working Papers 26080, National Bureau of Economic Research, Inc.
    12. Guido W. Imbens, 2022. "Causality in Econometrics: Choice vs Chance," Econometrica, Econometric Society, vol. 90(6), pages 2541-2566, November.
    13. Matthew Blackwell & Stefano Iacus & Gary King & Giuseppe Porro, 2009. "cem: Coarsened exact matching in Stata," Stata Journal, StataCorp LLC, vol. 9(4), pages 524-546, December.
    14. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    15. Duncan Chaplin & Arif Mamun & Ali Protik & John Schurrer & Divya Vohra & Kristine Bos & Hannah Burak & Laura Meyer & Anca Dumitrescu & Christopher Ksoll & Thomas Cook, "undated". "Grid Electricity Expansion in Tanzania by MCC: Findings from a Rigorous Impact Evaluation, Final Report," Mathematica Policy Research Reports 144768f69008442e96369195e, Mathematica Policy Research.
    16. Matthew A. Masten & Alexandre Poirier & Linqi Zhang, 2024. "Assessing Sensitivity to Unconfoundedness: Estimation and Inference," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 42(1), pages 1-13, January.
    17. Jens Ruhose & Stephan L. Thomsen & Insa Weilage, 2018. "The Wider Benefits of Adult Learning: Work-Related Training and Social Capital," CESifo Working Paper Series 7268, CESifo.
    18. Peter R. Mueser & Kenneth R. Troske & Alexey Gorislavsky, 2007. "Using State Administrative Data to Measure Program Performance," The Review of Economics and Statistics, MIT Press, vol. 89(4), pages 761-783, November.
    19. Timothy B. Armstrong & Michal Kolesár, 2021. "Finite‐Sample Optimal Estimation and Inference on Average Treatment Effects Under Unconfoundedness," Econometrica, Econometric Society, vol. 89(3), pages 1141-1177, May.
    20. Difang Huang & Jiti Gao & Tatsushi Oka, 2022. "Semiparametric Single-Index Estimation for Average Treatment Effects," Papers 2206.08503, arXiv.org, revised Jan 2025.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:46:y:2021:i:3:p:271-296. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.