IDEAS home Printed from https://ideas.repec.org/p/tse/wpaper/31798.html
   My bibliography  Save this paper

Should We Combine Difference In Differences with Conditioning on Pre-Treatment Outcomes?

Author

Listed:
  • Chabé-Ferret, Sylvain

Abstract

Applied researchers often combine Difference In Differences (DID) with conditioning on pre-treatment outcomes when the Parallel Trend Assumption (PTA) fails. I examine both the theoretical and empirical basis for this approach. I show that the theoretical argument that both methods combine their strengths – DID differencing out the permanent confounders while conditioning on pre-treatment outcomes captures the transitory ones – is incorrect. Worse, conditioning on pre-treatment outcomes might increase the bias of DID. Simulations of a realistic model of earnings dynamics and selection in a Job Training Program (JTP) show that this bias can be sizable in practice. Revisiting empirical studies comparing DID with RCTs, I also find that conditioning on pre-treatment outcomes increases the bias of DID. Taken together, these results suggest that we should not combine DID with conditioning on pre-treatment outcomes but rather use DID conditioning on covariates that are fixed over time. When the PTA fails, DID applied symmetrically around the treatment date performs well in simulations and when compared with RCTs. Matching on several observations of pre-treatment outcomes also performs well in simulations, but evidence on its empirical performance is lacking.

Suggested Citation

  • Chabé-Ferret, Sylvain, 2017. "Should We Combine Difference In Differences with Conditioning on Pre-Treatment Outcomes?," TSE Working Papers 17-824, Toulouse School of Economics (TSE).
    • Handle: RePEc:tse:wpaper:31798
    as

    Download full text from publisher

    File URL: https://www.tse-fr.eu/sites/default/files/TSE/documents/doc/wp/2017/wp_tse_824.pdf
    File Function: Full text
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Andrea Pufahl & Christoph R. Weiss, 2009. "Evaluating the effects of farm programmes: results from propensity score matching," European Review of Agricultural Economics, Oxford University Press and the European Agricultural and Applied Economics Publications Foundation, vol. 36(1), pages 79-101, March.
    2. Ashenfelter, Orley C, 1978. "Estimating the Effect of Training Programs on Earnings," The Review of Economics and Statistics, MIT Press, vol. 60(1), pages 47-57, February.
    3. Chabé-Ferret, Sylvain, 2015. "Analysis of the bias of Matching and Difference-in-Difference under alternative earnings and selection processes," Journal of Econometrics, Elsevier, vol. 185(1), pages 110-123.
    4. Andersson, Fredrik W. & Holzer, Harry J. & Lane, Julia & Rosenblum, David & Smith, Jeffrey A., 2013. "Does Federally-Funded Job Training Work? Nonexperimental Estimates of WIA Training Impacts Using Longitudinal Data on Workers and Firms," IZA Discussion Papers 7621, Institute of Labor Economics (IZA).
    5. Chabé-Ferret, Sylvain & Subervie, Julie, 2013. "How much green for the buck? Estimating additional and windfall effects of French agro-environmental schemes by DID-matching," Journal of Environmental Economics and Management, Elsevier, vol. 65(1), pages 12-27.
    6. Fatih Guvenen, 2007. "Learning Your Earning: Are Labor Income Shocks Really Very Persistent?," American Economic Review, American Economic Association, vol. 97(3), pages 687-712, June.
    7. James Heckman & Salvador Navarro-Lozano, 2004. "Using Matching, Instrumental Variables, and Control Functions to Estimate Economic Choice Models," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 30-57, February.
    8. Sebastian Galiani & Paul Gertler & Ernesto Schargrodsky, 2005. "Water for Life: The Impact of the Privatization of Water Services on Child Mortality," Journal of Political Economy, University of Chicago Press, vol. 113(1), pages 83-120, February.
    9. Alberto Abadie, 2005. "Semiparametric Difference-in-Differences Estimators," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 72(1), pages 1-19.
    10. Meredith Fowlie & Stephen P. Holland & Erin T. Mansur, 2012. "What Do Emissions Markets Deliver and to Whom? Evidence from Southern California's NOx Trading Program," American Economic Review, American Economic Association, vol. 102(2), pages 965-993, April.
    11. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
    12. Ashenfelter, Orley & Card, David, 1985. "Using the Longitudinal Structure of Earnings to Estimate the Effect of Training Programs," The Review of Economics and Statistics, MIT Press, vol. 67(4), pages 648-660, November.
    13. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-620, September.
    14. James J. Heckman & Hidehiko Ichimura & Petra E. Todd, 1997. "Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 605-654.
    15. Peter R. Mueser & Kenneth R. Troske & Alexey Gorislavsky, 2007. "Using State Administrative Data to Measure Program Performance," The Review of Economics and Statistics, MIT Press, vol. 89(4), pages 761-783, November.
    16. Rajeev H. Dehejia & Sadek Wahba, 2002. "Propensity Score-Matching Methods For Nonexperimental Causal Studies," The Review of Economics and Statistics, MIT Press, vol. 84(1), pages 151-161, February.
    17. MaCurdy, Thomas E., 1982. "The use of time series processes to model the error structure of earnings in a longitudinal data analysis," Journal of Econometrics, Elsevier, vol. 18(1), pages 83-114, January.
    18. Bassi, Laurie J, 1984. "Estimating the Effect of Training Programs with Non-Random Selection," The Review of Economics and Statistics, MIT Press, vol. 66(1), pages 36-43, February.
    19. Heckman, James J. & Lalonde, Robert J. & Smith, Jeffrey A., 1999. "The economics and econometrics of active labor market programs," Handbook of Labor Economics, in: O. Ashenfelter & D. Card (ed.), Handbook of Labor Economics, edition 1, volume 3, chapter 31, pages 1865-2097, Elsevier.
    20. Nickell, Stephen J, 1981. "Biases in Dynamic Models with Fixed Effects," Econometrica, Econometric Society, vol. 49(6), pages 1417-1426, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Matthieu Cassou & Julien Mousquès & Carine Franc, 2020. "General practitioners’ income and activity: the impact of multi-professional group practice in France," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 21(9), pages 1295-1315, December.
    2. Jeremias Nieminen & Ohto Kanninen & Hannu Karhunen, 2023. "The decentralization of public employment services and local governments’ responses to incentives," Journal of Economic Geography, Oxford University Press, vol. 23(6), pages 1371-1395.
    3. Meyer, Maximilian & Klingelhoeffer, Ekkehard & Naidoo, Robin & Wingate, Vladimir & Börner, Jan, 2021. "Tourism opportunities drive woodland and wildlife conservation outcomes of community-based conservation in Namibia's Zambezi region," Ecological Economics, Elsevier, vol. 180(C).
    4. Leopoldo Fergusson & Juan Felipe Riaño & B.K. Song, 2020. "Media, Secret Ballot and Democratization in the US," Documentos de Trabajo 18252, The Latin American and Caribbean Economic Association (LACEA).
    5. Hetschko, Clemens & Schöb, Ronnie & Wolf, Tobias, 2020. "Income support, employment transitions and well-being," Labour Economics, Elsevier, vol. 66(C).
    6. Cesare Righi & Timothy Simcoe, 2022. "Patenting inventions or inventing patents? Continuation practice at the USPTO," Economics Working Papers 1820, Department of Economics and Business, Universitat Pompeu Fabra.
    7. Bührle, Anna Theresa, 2021. "Do tax loss restrictions distort venture capital funding of start-ups?," ZEW Discussion Papers 21-008, ZEW - Leibniz Centre for European Economic Research.
    8. Uehleke, Reinhard & Petrick, Martin & Hüttel, Silke, 2022. "Evaluations of agri-environmental schemes based on observational farm data: The importance of covariate selection," Land Use Policy, Elsevier, vol. 114(C).
    9. Srhoj, Stjepan & Kovač, Dejan & Shapiro, Jacob N. & Filer, Randall K., 2023. "The impact of delay: Evidence from formal out-of-court restructuring," Journal of Corporate Finance, Elsevier, vol. 78(C).
    10. Gallego, Jorge & Maldonado, Stanislao & Trujillo, Lorena, 2020. "From curse to blessing? institutional reform and resource booms in Colombia," Journal of Economic Behavior & Organization, Elsevier, vol. 178(C), pages 174-193.
    11. Raphaël CHIAPPINI & Sophie POMMET, 2023. "The impact of public support for innovation on SME performance and efficiency," Bordeaux Economics Working Papers 2023-06, Bordeaux School of Economics (BSE).
    12. Stjepan Srhoj & Michal Lapinski & Janette Walde, 2021. "Impact evaluation of business development grants on SME performance," Small Business Economics, Springer, vol. 57(3), pages 1285-1301, October.
    13. Ohto Kanninen & Hannu Karhunen & Jeremias Nieminen, "undated". "Effect of Secondary Education on Cognitive and Non-cognitive Skills," Working Papers 337, Työn ja talouden tutkimus LABORE, The Labour Institute for Economic Research LABORE.
    14. J. Michael McWilliams & Laura A. Hatfield & Bruce E. Landon & Michael E. Chernew, 2019. "Spending Reductions in the Medicare Shared Savings Program: Selection or Savings?," NBER Working Papers 26403, National Bureau of Economic Research, Inc.
    15. Dvouletý Ondřej & Čadil Jan & Mirošník Karel, 2019. "Do Firms Supported by Credit Guarantee Schemes Report Better Financial Results 2 Years After the End of Intervention?," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 19(1), pages 1-20, January.
    16. Cesare Righi & Timothy Simcoe, 2020. "Patenting Inventions or Inventing Patents? Continuation Practice at the USPTO," NBER Working Papers 27686, National Bureau of Economic Research, Inc.
    17. Rob Kuijpers, 2019. "Value Chain Development as Public Policy: Conceptualization and Evidence from the Agri-Food Sector in Bangladesh," LICOS Discussion Papers 41419, LICOS - Centre for Institutions and Economic Performance, KU Leuven.
    18. Raschid Amamou & Áron Gereben & Marcin Wolski, 2023. "Assessing the impact of the EIB’s intermediated lending to SMEs during funding shocks," Small Business Economics, Springer, vol. 60(3), pages 975-1007, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chabé-Ferret, Sylvain, 2015. "Analysis of the bias of Matching and Difference-in-Difference under alternative earnings and selection processes," Journal of Econometrics, Elsevier, vol. 185(1), pages 110-123.
    2. Richard Blundell & Monica Costa Dias, 2009. "Alternative Approaches to Evaluation in Empirical Microeconomics," Journal of Human Resources, University of Wisconsin Press, vol. 44(3).
    3. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    4. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    5. Peter R. Mueser & Kenneth R. Troske & Alexey Gorislavsky, 2007. "Using State Administrative Data to Measure Program Performance," The Review of Economics and Statistics, MIT Press, vol. 89(4), pages 761-783, November.
    6. Jones A.M & Rice N, 2009. "Econometric Evaluation of Health Policies," Health, Econometrics and Data Group (HEDG) Working Papers 09/09, HEDG, c/o Department of Economics, University of York.
    7. Lechner, Michael & Wunsch, Conny, 2013. "Sensitivity of matching-based program evaluations to the availability of control variables," Labour Economics, Elsevier, vol. 21(C), pages 111-121.
    8. Chabé-Ferret, Sylvain, 2012. "Matching vs Differencing when Estimating Treatment Effects with Panel Data: the Example of the Effect of Job Training Programs on Earnings," LERNA Working Papers 12.24.381, LERNA, University of Toulouse.
    9. Regner, Hakan, 2002. "A nonexperimental evaluation of training programs for the unemployed in Sweden," Labour Economics, Elsevier, vol. 9(2), pages 187-206, April.
    10. Flores, Carlos A. & Mitnik, Oscar A., 2009. "Evaluating Nonexperimental Estimators for Multiple Treatments: Evidence from Experimental Data," IZA Discussion Papers 4451, Institute of Labor Economics (IZA).
    11. Jens Ruhose & Stephan L. Thomsen & Insa Weilage, 2018. "The Wider Benefits of Adult Learning: Work-Related Training and Social Capital," CESifo Working Paper Series 7268, CESifo.
    12. Jorge Rodríguez & Fernando Saltiel & Sergio Urzúa, 2022. "Dynamic treatment effects of job training," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(2), pages 242-269, March.
    13. James J. Heckman & Jeffrey A. Smith, 1999. "The Pre-Program Earnings Dip and the Determinants of Participation in a Social Program: Implications for Simple Program Evaluation Strategies," NBER Working Papers 6983, National Bureau of Economic Research, Inc.
    14. Gueorgui Kambourov & Iourii Manovskii & Miana Plesca, 2020. "Occupational mobility and the returns to training," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 53(1), pages 174-211, February.
    15. Gustavo Canavire-Bacarreza & Luis Castro Peñarrieta & Darwin Ugarte Ontiveros, 2021. "Outliers in Semi-Parametric Estimation of Treatment Effects," Econometrics, MDPI, vol. 9(2), pages 1-32, April.
    16. Kluve, Jochen & Lehmann, Hartmut & Schmidt, Christoph M., 2008. "Disentangling Treatment Effects of Active Labor Market Policies: The Role of Labor Force Status Sequences," Labour Economics, Elsevier, vol. 15(6), pages 1270-1295, December.
    17. Centeno, Luis & Centeno, Mário & Novo, Álvaro A., 2009. "Evaluating job-search programs for old and young individuals: Heterogeneous impact on unemployment duration," Labour Economics, Elsevier, vol. 16(1), pages 12-25, January.
    18. Ruhose, Jens & Thomsen, Stephan, 2017. "Non-Monetary Benefits of Continuous Training," VfS Annual Conference 2017 (Vienna): Alternative Structures for Money and Banking 168169, Verein für Socialpolitik / German Economic Association.
    19. Jens Ruhose & Stephan L. Thomsen & Insa Weilage, 2018. "The Wider Benefits of Adult Learning: Work-Related Training and Social Capital," SOEPpapers on Multidisciplinary Panel Data Research 1004, DIW Berlin, The German Socio-Economic Panel (SOEP).
    20. Hara, Hiromi, 2022. "The effect of public-sponsored job training in Japan," Journal of the Japanese and International Economies, Elsevier, vol. 64(C).

    More about this item

    Keywords

    Difference in Differences - Matching - Selection Model - Treatment Effects;

    JEL classification:

    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • C23 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Models with Panel Data; Spatio-temporal Models

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:tse:wpaper:31798. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/tsetofr.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.