IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp14682.html
   My bibliography  Save this paper

COVID-19 Mobility Policies Impacts: How Credible Are Difference-in-Differences Estimates?

Author

Listed:
  • Weill, Joakim A.

    (University of California, Davis)

  • Stigler, Matthieu

    (Stanford University)

  • Deschenes, Olivier

    (University of California, Santa Barbara)

  • Springborn, Michael R.

    (University of California, Davis)

Abstract

The COVID-19 pandemic brought unprecedented policy responses and a large literature evaluating their impacts. This paper re-examines and add to the evidence on the impact of COVID-19 mobility-restricting policies on mobility indicators. We first find that two-way fixed effects estimates are not robust to minor specification changes, where the same policy can be found to significantly increase or decrease mobility, depending on the specification. Therefore, due to the large number of researcher's degrees-of-exibility, researchers can focus on a set of results that appears stable, while ignoring problematic ones. Further, recently developed heterogeneity-robust difference-in-differences methods only partially mitigate these issues.

Suggested Citation

  • Weill, Joakim A. & Stigler, Matthieu & Deschenes, Olivier & Springborn, Michael R., 2021. "COVID-19 Mobility Policies Impacts: How Credible Are Difference-in-Differences Estimates?," IZA Discussion Papers 14682, Institute of Labor Economics (IZA).
    • Handle: RePEc:iza:izadps:dp14682
    as

    Download full text from publisher

    File URL: https://docs.iza.org/dp14682.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    2. Allcott, Hunt & Boxell, Levi & Conway, Jacob & Gentzkow, Matthew & Thaler, Michael & Yang, David, 2020. "Polarization and public health: Partisan differences in social distancing during the coronavirus pandemic," Journal of Public Economics, Elsevier, vol. 191(C).
    3. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    4. Villas-Boas, Sofia B & Sears, James & Villas-Boas, Miguel & Villas-Boas, Vasco, 2020. "Are We #StayingHome to Flatten the Curve?," Department of Agricultural & Resource Economics, UC Berkeley, Working Paper Series qt5h97n884, Department of Agricultural & Resource Economics, UC Berkeley.
    5. Michael A. Clemens & Jennifer Hunt, 2019. "The Labor Market Effects of Refugee Waves: Reconciling Conflicting Results," ILR Review, Cornell University, ILR School, vol. 72(4), pages 818-857, August.
    6. Rebucci, Alessandro & Elenev, Vadim & , & Simeonova, Emilia, 2021. "Direct and Spillover Effects from Staggered Adoption of Health Policies: Evidence from Covid-19 Stay-at-Home Orders," CEPR Discussion Papers 16649, C.E.P.R. Discussion Papers.
    7. Dhaval Dave & Andrew I. Friedson & Kyutaro Matsuzawa & Joseph J. Sabia, 2021. "When Do Shelter‐In‐Place Orders Fight Covid‐19 Best? Policy Heterogeneity Across States And Adoption Time," Economic Inquiry, Western Economic Association International, vol. 59(1), pages 29-52, January.
    8. Joakim A. Weill & Matthieu Stigler & Olivier Deschenes & Michael R. Springborn, 2020. "Opinion: Social distancing responses to COVID-19 emergency declarations strongly differentiated by income," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 117(33), pages 19658-19660, August.
    9. Sumedha Gupta & Thuy D. Nguyen & Felipe Lozano Rojas & Shyam Raman & Byungkyu Lee & Ana Bento & Kosali I. Simon & Coady Wing, 2020. "Tracking Public and Private Responses to the COVID-19 Epidemic: Evidence from State and Local Government Actions," NBER Working Papers 27027, National Bureau of Economic Research, Inc.
    10. Sun, Liyang & Abraham, Sarah, 2021. "Estimating dynamic treatment effects in event studies with heterogeneous treatment effects," Journal of Econometrics, Elsevier, vol. 225(2), pages 175-199.
    11. Goolsbee, Austan & Syverson, Chad, 2021. "Fear, lockdown, and diversion: Comparing drivers of pandemic economic decline 2020," Journal of Public Economics, Elsevier, vol. 193(C).
    12. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    13. David Holtz & Michael Zhao & Seth G. Benzell & Cathy Y. Cao & Mohammad Amin Rahimian & Jeremy Yang & Jennifer Allen & Avinash Collis & Alex Moehring & Tara Sowrirajan & Dipayan Ghosh & Yunhao Zhang & , 2020. "Interdependence and the cost of uncoordinated responses to COVID-19," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 117(33), pages 19837-19843, August.
    14. Jonathan Roth & Pedro H. C. Sant'Anna, 2023. "When Is Parallel Trends Sensitive to Functional Form?," Econometrica, Econometric Society, vol. 91(2), pages 737-747, March.
    15. Christopher L. Foote & Christopher F. Goetz, 2008. "The Impact of Legalized Abortion on Crime: Comment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 123(1), pages 407-423.
    16. Maximilian Kasy, 2021. "Of Forking Paths and Tied Hands: Selective Publication of Findings, and What Economists Should Do about It," Journal of Economic Perspectives, American Economic Association, vol. 35(3), pages 175-192, Summer.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Giulia Bettin & Isabella Giorgetti & Stefano Staffolani, 2024. "The impact of Covid-19 lockdown on the gender gap in the Italian labour market," Review of Economics of the Household, Springer, vol. 22(1), pages 1-33, March.
    2. Aditya Kulkarni & Minkyong Kim & Jayanta Bhattacharya & Joydeep Bhattacharya, 2023. "Businesses in high-income zip codes often saw sharper visit reductions during the COVID-19 pandemic," Palgrave Communications, Palgrave Macmillan, vol. 10(1), pages 1-10, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Joakim A. Weill & Matthieu Stigler & Olivier Deschenes & Michael R. Springborn, 2021. "Researchers' Degrees-of-Flexibility and the Credibility of Difference-in-Differences Estimates: Evidence From the Pandemic Policy Evaluations," NBER Working Papers 29550, National Bureau of Economic Research, Inc.
    2. Hunt Allcott & Levi Boxell & Jacob C. Conway & Billy A. Ferguson & Matthew Gentzkow & Benjamin Goldman, 2020. "What Explains Temporal and Geographic Variation in the Early US Coronavirus Pandemic?," NBER Working Papers 27965, National Bureau of Economic Research, Inc.
    3. Sumedha Gupta & Kosali I. Simon & Coady Wing, 2020. "Mandated and Voluntary Social Distancing During The COVID-19 Epidemic: A Review," NBER Working Papers 28139, National Bureau of Economic Research, Inc.
    4. Miriam Marcén & Marina Morales, 2021. "The intensity of COVID‐19 nonpharmaceutical interventions and labor market outcomes in the public sector," Journal of Regional Science, Wiley Blackwell, vol. 61(4), pages 775-798, September.
    5. Andrew Goodman-Bacon & Jan Marcus, 2020. "Difference-in-Differences to Identify Causal Effects of COVID-19 Policies," Discussion Papers of DIW Berlin 1870, DIW Berlin, German Institute for Economic Research.
    6. Bisin, Alberto & Moro, Andrea, 2022. "JUE insight: Learning epidemiology by doing: The empirical implications of a Spatial-SIR model with behavioral responses," Journal of Urban Economics, Elsevier, vol. 127(C).
    7. Diane Alexander & Ezra Karger, 2023. "Do Stay-at-Home Orders Cause People to Stay at Home? Effects of Stay-at-Home Orders on Consumer Behavior," The Review of Economics and Statistics, MIT Press, vol. 105(4), pages 1017-1027, July.
    8. Herby, Jonas & Jonung, Lars & Hanke, Steve, 2022. "A Literature Review and Meta-Analysis of the Effects of Lockdowns on Covid-19 Mortality - II," MPRA Paper 113732, University Library of Munich, Germany.
    9. Edward L. Glaeser & Ginger Z. Jin & Benjamin T. Leyden & Michael Luca, 2021. "Learning from deregulation: The asymmetric impact of lockdown and reopening on risky behavior during COVID‐19," Journal of Regional Science, Wiley Blackwell, vol. 61(4), pages 696-709, September.
    10. Christopher J. Cronin & William N. Evans, 2020. "Private Precaution and Public Restrictions: What Drives Social Distancing and Industry Foot Traffic in the COVID-19 Era?," NBER Working Papers 27531, National Bureau of Economic Research, Inc.
    11. Janssen, Aljoscha & Shapiro, Matthew, 2020. "Does Precise Case Information Limit Precautionary Behavior? Evidence from COVID-19 in Singapore," Working Paper Series 1344, Research Institute of Industrial Economics.
    12. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    13. Gonzalez-Eiras, Martín & Niepelt, Dirk, 2022. "The political economy of early COVID-19 interventions in US states," Journal of Economic Dynamics and Control, Elsevier, vol. 140(C).
    14. Krzysztof Zaremba, 2023. "Opening of hotels and ski facilities: Impact on mobility, spending, and Covid‐19 outcomes," Health Economics, John Wiley & Sons, Ltd., vol. 32(5), pages 1148-1180, May.
    15. M. Kate Bundorf & Jill DeMatteis & Grant Miller & Maria Polyakova & Jialu L. Streeter & Jonathan Wivagg, 2021. "Risk Perceptions and Protective Behaviors: Evidence from COVID-19 Pandemic," NBER Working Papers 28741, National Bureau of Economic Research, Inc.
    16. Cook, Jonathan & Newberger, Noah & Smalling, Sami, 2020. "The spread of social distancing," Economics Letters, Elsevier, vol. 196(C).
    17. Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023. "Do economists replicate?," Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 219-232.
    18. Callaway, Brantly & Li, Tong, 2023. "Policy evaluation during a pandemic," Journal of Econometrics, Elsevier, vol. 236(1).
    19. Martin Andersen & Johanna Catherine Maclean & Michael F. Pesko & Kosali I. Simon, 2020. "Paid sick-leave and physical mobility: Evidence from the United States during a pandemic," NBER Working Papers 27138, National Bureau of Economic Research, Inc.
    20. Shin, Su Hyun & Ji, Hyunjung & Lim, HanNa, 2021. "Heterogeneity in preventive behaviors during COVID-19: Health risk, economic insecurity, and slanted information," Social Science & Medicine, Elsevier, vol. 278(C).

    More about this item

    Keywords

    COVID-19; social distancing; mobility; difference-in-differences; researcher degrees-of-flexibility; mobility-restricting policies;
    All these keywords.

    JEL classification:

    • I12 - Health, Education, and Welfare - - Health - - - Health Behavior
    • I18 - Health, Education, and Welfare - - Health - - - Government Policy; Regulation; Public Health
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • C23 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Models with Panel Data; Spatio-temporal Models
    • H75 - Public Economics - - State and Local Government; Intergovernmental Relations - - - State and Local Government: Health, Education, and Welfare

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp14682. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Holger Hinte (email available below). General contact details of provider: https://edirc.repec.org/data/izaaade.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.