IDEAS home Printed from https://ideas.repec.org/a/bpj/jecome/v7y2018i1p16n9.html
   My bibliography  Save this article

Inference with Difference-in-Differences Revisited

Author

Listed:
  • Brewer Mike

    (Institute for Social and Economic Research, University of Essex, Colchester, Essex CO4 3SQ, UK)

  • Crossley Thomas F.

    () (Department of Economics, University of Essex, Colchester, Essex CO4 3SQ, UK)

  • Joyce Robert

    (Institute for Fiscal Studies, London WC1E 7AE, UK)

Abstract

A growing literature on inference in difference-in-differences (DiD) designs has been pessimistic about obtaining hypothesis tests of the correct size, particularly with few groups. We provide Monte Carlo evidence for four points: (i) it is possible to obtain tests of the correct size even with few groups, and in many settings very straightforward methods will achieve this; (ii) the main problem in DiD designs with grouped errors is instead low power to detect real effects; (iii) feasible GLS estimation combined with robust inference can increase power considerably whilst maintaining correct test size – again, even with few groups, and (iv) using OLS with robust inference can lead to a perverse relationship between power and panel length.

Suggested Citation

  • Brewer Mike & Crossley Thomas F. & Joyce Robert, 2018. "Inference with Difference-in-Differences Revisited," Journal of Econometric Methods, De Gruyter, vol. 7(1), pages 1-16, January.
  • Handle: RePEc:bpj:jecome:v:7:y:2018:i:1:p:16:n:9
    as

    Download full text from publisher

    File URL: https://www.degruyter.com/view/j/jem.2018.7.issue-1/jem-2017-0005/jem-2017-0005.xml?format=INT
    Download Restriction: For access to full text, subscription to the journal or payment for the individual article is required.

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Guido W. Imbens & Michal Kolesár, 2016. "Robust Standard Errors in Small Samples: Some Practical Advice," The Review of Economics and Statistics, MIT Press, vol. 98(4), pages 701-712, October.
    2. A. Colin Cameron & Jonah B. Gelbach & Douglas L. Miller, 2008. "Bootstrap-Based Improvements for Inference with Clustered Errors," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 414-427, August.
    3. James G. MacKinnon & Matthew D. Webb, 2017. "Wild Bootstrap Inference for Wildly Different Cluster Sizes," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(2), pages 233-254, March.
    4. Matthew D. Webb, 2014. "Reworking Wild Bootstrap Based Inference For Clustered Errors," Working Paper 1315, Economics Department, Queen's University.
    5. Thomas Barrios & Rebecca Diamond & Guido W. Imbens & Michal Kolesár, 2012. "Clustering, Spatial Correlations, and Randomization Inference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 107(498), pages 578-591, June.
    6. Jeffrey M. Wooldridge, 2003. "Cluster-Sample Methods in Applied Econometrics," American Economic Review, American Economic Association, vol. 93(2), pages 133-138, May.
    7. Bester, C. Alan & Conley, Timothy G. & Hansen, Christian B., 2011. "Inference with dependent data using cluster covariance estimators," Journal of Econometrics, Elsevier, vol. 165(2), pages 137-151.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Mathias Huebener & Jan Marcus, 2015. "Moving up a Gear: The Impact of Compressing Instructional Time into Fewer Years of Schooling," Discussion Papers of DIW Berlin 1450, DIW Berlin, German Institute for Economic Research.
    2. repec:aea:aecrev:v:107:y:2017:i:5:p:168-73 is not listed on IDEAS
    3. Hollingsworth, Bruce & Ohinata, Asako & Picchio, Matteo & Walker, Ian, 2017. "Labour supply and informal care supply: The impacts of financial support for long-term elderly care," GLO Discussion Paper Series 118, Global Labor Organization (GLO).
    4. Congdon Fors, Heather & Houngbedji, Kenneth & Lindskog, Annika, 2019. "Land certification and schooling in rural Ethiopia," World Development, Elsevier, vol. 115(C), pages 190-208.
    5. Georg Graetz & Guy Michaels, 2017. "Is Modern Technology Responsible for Jobless Recoveries?," American Economic Review, American Economic Association, vol. 107(5), pages 168-173, May.
    6. Bruno Ferman & Cristine Pinto, 2019. "Inference in Differences-in-Differences with Few Treated Groups and Heteroskedasticity," The Review of Economics and Statistics, MIT Press, vol. 101(3), pages 452-467, July.
    7. Noémi Kreif & Richard Grieve & Dominik Hangartner & Alex James Turner & Silviya Nikolova & Matt Sutton, 2016. "Examination of the Synthetic Control Method for Evaluating Health Policies with Multiple Treated Units," Health Economics, John Wiley & Sons, Ltd., vol. 25(12), pages 1514-1528, December.
    8. Asako OHINATA & Matteo PICCHIO, 2015. "The Financial Support for Long-Term Elderly Care and Household Savings Behaviour," Working Papers 411, Universita' Politecnica delle Marche (I), Dipartimento di Scienze Economiche e Sociali.
    9. Weber, Sylvain & Puddu, Stefano & Pacheco, Diana, 2017. "Move it! How an electric contest motivates households to shift their load profile," Energy Economics, Elsevier, vol. 68(C), pages 255-270.
    10. Verdugo, Gregory, 2016. "Real wage cyclicality in the Eurozone before and during the Great Recession: Evidence from micro data," European Economic Review, Elsevier, vol. 82(C), pages 46-69.
    11. Brewer, Mike & Crossley, Thomas F. & Zilio, Federico, 2019. "What Do We Really Know about the Employment Effects of the UK's National Minimum Wage?," IZA Discussion Papers 12369, Institute of Labor Economics (IZA).
    12. repec:fgv:eesptd:411 is not listed on IDEAS
    13. Advani, Arun & Sloczynski, Tymon, 2013. "Mostly Harmless Simulations? On the Internal Validity of Empirical Monte Carlo Studies," IZA Discussion Papers 7874, Institute of Labor Economics (IZA).
    14. Li, Qing & Sweetman, Arthur, 2014. "The quality of immigrant source country educational outcomes: Do they matter in the receiving country?," Labour Economics, Elsevier, vol. 26(C), pages 81-93.
    15. Drange, Nina & Telle, Kjetil, 2015. "Promoting integration of immigrants: Effects of free child care on child enrollment and parental employment," Labour Economics, Elsevier, vol. 34(C), pages 26-38.
    16. repec:zbw:espost:162940 is not listed on IDEAS
    17. Slawa Rokicki & Jessica Cohen & Gunther Fink & Joshua Salomon & Mary Beth Landrum, 2018. "Inference with difference-in-differences with a small number of groups: a review, simulation study and empirical application using SHARE data," CHaRMS Working Papers 18-01, Centre for HeAlth Research at the Management School (CHaRMS).
    18. Thomas C. Buchmueller & Colleen Carey, 2018. "The Effect of Prescription Drug Monitoring Programs on Opioid Utilization in Medicare," American Economic Journal: Economic Policy, American Economic Association, vol. 10(1), pages 77-112, February.
    19. Geoffrey R. Dunbar, 2014. "Demographics and the Demand for Currency," Staff Working Papers 14-59, Bank of Canada.
    20. repec:bla:econom:v:85:y:2018:i:340:p:701-734 is not listed on IDEAS
    21. Cockx, Bart & Ghirelli, Corinna, 2016. "Scars of recessions in a rigid labor market," Labour Economics, Elsevier, vol. 41(C), pages 162-176.
    22. repec:eee:ecoedu:v:58:y:2017:i:c:p:1-14 is not listed on IDEAS

    More about this item

    Keywords

    cluster robust; difference in differences; feasible GLS; hypothesis test; power;

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C13 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Estimation: General
    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bpj:jecome:v:7:y:2018:i:1:p:16:n:9. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Peter Golla). General contact details of provider: https://www.degruyter.com .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.