IDEAS home Printed from https://ideas.repec.org/p/qed/wpaper/1387.html
   My bibliography  Save this paper

Pitfalls when Estimating Treatment Effects Using Clustered Data

Author

Listed:
  • James G. MacKinnon

    () (Queen's University)

  • Matthew D. Webb

    () (Carleton University)

Abstract

Inference for estimates of treatment effects with clustered data requires great care when treatment is assigned at the group level. This is true for both pure treatment models and difference-in-differences regressions. Even when the number of clusters is quite large, cluster-robust standard errors can be much too small if the number of treated (or control) clusters is small. Standard errors also tend to be too small when cluster sizes vary a lot, resulting in too many false positives. Bootstrap methods generally perform better than t-tests, but they can also yield very misleading inferences in some cases.

Suggested Citation

  • James G. MacKinnon & Matthew D. Webb, 2017. "Pitfalls when Estimating Treatment Effects Using Clustered Data," Working Papers 1387, Queen's University, Department of Economics.
  • Handle: RePEc:qed:wpaper:1387
    as

    Download full text from publisher

    File URL: http://qed.econ.queensu.ca/working_papers/papers/qed_wp_1387.pdf
    File Function: First version 2017
    Download Restriction: no

    References listed on IDEAS

    as
    1. James G. MacKinnon & Matthew D. Webb, 2016. "Randomization Inference for Difference-in-Differences with Few Treated Clusters," Carleton Economic Papers 16-11, Carleton University, Department of Economics.
    2. Guido W. Imbens & Michal Kolesár, 2016. "Robust Standard Errors in Small Samples: Some Practical Advice," The Review of Economics and Statistics, MIT Press, vol. 98(4), pages 701-712, October.
    3. repec:clg:wpaper:2013-20 is not listed on IDEAS
    4. James G. MacKinnon & Matthew D. Webb, 2017. "The Wild Bootstrap for Few (Treated) Clusters," Working Papers 1364, Queen's University, Department of Economics.
    5. Ferman, Bruno & Pinto, Cristine Campos de Xavier, 2015. "Inference in differences-in-differences with few treated groups and heteroskedasticity," Textos para discussão 406, FGV/EESP - Escola de Economia de São Paulo, Getulio Vargas Foundation (Brazil).
    6. A. Colin Cameron & Jonah B. Gelbach & Douglas L. Miller, 2008. "Bootstrap-Based Improvements for Inference with Clustered Errors," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 414-427, August.
    7. Davidson, Russell & Flachaire, Emmanuel, 2008. "The wild bootstrap, tamed at last," Journal of Econometrics, Elsevier, vol. 146(1), pages 162-169, September.
    8. Timothy G. Conley & Christopher R. Taber, 2011. "Inference with "Difference in Differences" with a Small Number of Policy Changes," The Review of Economics and Statistics, MIT Press, vol. 93(1), pages 113-125, February.
    9. repec:tpr:restat:v:99:y:2017:i:4:p:698-709 is not listed on IDEAS
    10. Davidson, Russell & MacKinnon, James G., 1999. "The Size Distortion Of Bootstrap Tests," Econometric Theory, Cambridge University Press, vol. 15(03), pages 361-376, June.
    11. Russell Davidson & James MacKinnon, 2000. "Bootstrap tests: how many bootstraps?," Econometric Reviews, Taylor & Francis Journals, vol. 19(1), pages 55-68.
    12. A. Colin Cameron & Douglas L. Miller, 2015. "A Practitioner’s Guide to Cluster-Robust Inference," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 317-372.
    13. repec:wly:japmet:v:32:y:2017:i:2:p:233-254 is not listed on IDEAS
    14. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, Oxford University Press, vol. 119(1), pages 249-275.
    15. MacKinnon, James G. & Orregaard Nielsen, Morten & Webb, Matthew D., 2017. "Bootstrap and Asymptotic Inference with Multiway Clustering," Queen's Economics Department Working Papers 274712, Queen's University - Department of Economics.
    16. Matthew D. Webb, 2014. "Reworking Wild Bootstrap Based Inference for Clustered Errors," Working Papers 1315, Queen's University, Department of Economics.
    17. MacKinnon, James G. & White, Halbert, 1985. "Some heteroskedasticity-consistent covariance matrix estimators with improved finite sample properties," Journal of Econometrics, Elsevier, vol. 29(3), pages 305-325, September.
    18. Bester, C. Alan & Conley, Timothy G. & Hansen, Christian B., 2011. "Inference with dependent data using cluster covariance estimators," Journal of Econometrics, Elsevier, vol. 165(2), pages 137-151.
    19. James G. Mackinnon & Matthew D. Webb, 2017. "Wild Bootstrap Inference for Wildly Different Cluster Sizes," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(2), pages 233-254, March.
    Full references (including those not matched with items on IDEAS)

    More about this item

    Keywords

    CRVE; grouped data; clustered data; panel data; wild cluster bootstrap; difference-in-differences; DiD regression;

    JEL classification:

    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General
    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • C23 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Models with Panel Data; Spatio-temporal Models

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:qed:wpaper:1387. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Mark Babcock). General contact details of provider: http://edirc.repec.org/data/qedquca.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.