IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v36y2012i6p449-474.html
   My bibliography  Save this article

Bounding the Effects of Social Experiments

Author

Listed:
  • Jeffrey Grogger

Abstract

Background: Social experiments frequently exploit data from administrative records. However, most administrative data systems are designed to track earnings or benefit payments among residents within a single state. When an experimental participant moves across state lines, his entries in the data system of his state of origin consist entirely of zeros. Such attrition may bias the estimated effect of the experiment. Objective: To estimate the attrition arising from interstate mobility and provide bounds on the effect of the experiment. Method: Attrition is estimated from runs of zeros at the end of the sample period. Bounds are constructed from these estimates. These estimates can be refined by imposing a stationarity assumption. Results: The width of the estimated bounds depends importantly on the nature of the data being analyzed. Negatively correlated outcomes provide tighter bounds than positively correlated outcomes. Conclusion: Attrition can introduce considerable ambiguity into the estimated effects of experimental programs. To reduce ambiguity, one should collect as much data as possible. Even data on outcomes of no direct interest to the objectives of the experiment may be valuable for reducing the ambiguity that arises due to attrition.

Suggested Citation

  • Jeffrey Grogger, 2012. "Bounding the Effects of Social Experiments," Evaluation Review, , vol. 36(6), pages 449-474, December.
    • Handle: RePEc:sae:evarev:v:36:y:2012:i:6:p:449-474
    DOI: 10.1177/0193841X13482125
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X13482125
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X13482125?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Marianne P. Bitler & Jonah B. Gelbach & Hilary W. Hoynes, 2006. "What Mean Impacts Miss: Distributional Effects of Welfare Reform Experiments," American Economic Review, American Economic Association, vol. 96(4), pages 988-1012, September.
    2. Howard S. Bloom & Larry L. Orr & Stephen H. Bell & George Cave & Fred Doolittle & Winston Lin & Johannes M. Bos, 1997. "The Benefits and Costs of JTPA Title II-A Programs: Key Findings from the National Job Training Partnership Act Study," Journal of Human Resources, University of Wisconsin Press, vol. 32(3), pages 549-576.
    3. Greenberg, David & Moffitt, Robert & Friedmann, John, 1981. "Underreporting and Experimental Effects on Work Effort: Evidence from the Gary Income Maintenance Experiment," The Review of Economics and Statistics, MIT Press, vol. 63(4), pages 581-589, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Robert Collinson & John Eric Humphries & Nicholas Mader & Davin Reed & Daniel Tannenbaum & Winnie van Dijk, 2024. "Eviction and Poverty in American Cities," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 139(1), pages 57-120.
    2. Veronica Minaya & Judith Scott-Clayton, 2018. "Labor Market Outcomes and Postsecondary Accountability: Are Imperfect Metrics Better Than None?," NBER Chapters, in: Productivity in Higher Education, pages 67-104, National Bureau of Economic Research, Inc.
    3. Veronica Minaya & Judith Scott-Clayton, 2016. "Labor Market Outcomes and Postsecondary Accountability: Are Imperfect Metrics Better than None?," NBER Working Papers 22880, National Bureau of Economic Research, Inc.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Toru Kitagawa & Aleksey Tetenov, 2021. "Equality-Minded Treatment Choice," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(2), pages 561-574, March.
    2. David Card & Pablo Ibarrarán & Ferdinando Regalia & David Rosas-Shady & Yuri Soares, 2011. "The Labor Market Impacts of Youth Training in the Dominican Republic," Journal of Labor Economics, University of Chicago Press, vol. 29(2), pages 267-300.
    3. Jesse Rothstein & Till von Wachter, 2016. "Social Experiments in the Labor Market," NBER Working Papers 22585, National Bureau of Economic Research, Inc.
    4. Toru Kitagawa & Aleksey Tetenov, 2017. "Equality-minded treatment choice," CeMMAP working papers 10/17, Institute for Fiscal Studies.
    5. Alan B. Krueger, 2002. "Inequality, Too Much of a Good Thing," Working Papers 845, Princeton University, Department of Economics, Industrial Relations Section..
    6. Pedro H. C. Sant'Anna & Xiaojun Song & Qi Xu, 2022. "Covariate distribution balance via propensity scores," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(6), pages 1093-1120, September.
    7. repec:ist:iujspc:v:0:y:2022:i:83:p:101-140 is not listed on IDEAS
    8. Alexander Gelber & Adam Isen & Judd B. Kessler, 2014. "The Effects of Youth Employment: Evidence from New York City Summer Youth Employment Program Lotteries," NBER Working Papers 20810, National Bureau of Economic Research, Inc.
    9. Undral Byambadalai & Tatsushi Oka & Shota Yasui, 2024. "Estimating Distributional Treatment Effects in Randomized Experiments: Machine Learning for Variance Reduction," Papers 2407.16037, arXiv.org.
    10. Ronald D'Amico & Peter Z. Schochet, "undated". "The Evaluation of the Trade Adjustment Assistance Program: A Synthesis of Major Findings," Mathematica Policy Research Reports c6b34445ad854f5d8178f580f, Mathematica Policy Research.
    11. Bruno Crépon & Gerard J. van den Berg, 2016. "Active Labor Market Policies," Annual Review of Economics, Annual Reviews, vol. 8(1), pages 521-546, October.
    12. Maier, Michael, 2011. "Tests for distributional treatment effects under unconfoundedness," Economics Letters, Elsevier, vol. 110(1), pages 49-51, January.
    13. Firpo, Sergio & Galvao, Antonio F. & Kobus, Martyna & Parker, Thomas & Rosa-Dias, Pedro, 2020. "Loss Aversion and the Welfare Ranking of Policy Interventions," IZA Discussion Papers 13176, Institute of Labor Economics (IZA).
    14. Guanyi Wang, 2024. "Robust Network Targeting with Multiple Nash Equilibria," Papers 2410.20860, arXiv.org, revised Nov 2024.
    15. Sloczynski, Tymon, 2018. "A General Weighted Average Representation of the Ordinary and Two-Stage Least Squares Estimands," IZA Discussion Papers 11866, Institute of Labor Economics (IZA).
    16. Markus Frölich & Blaise Melly, 2013. "Identification of Treatment Effects on the Treated with One-Sided Non-Compliance," Econometric Reviews, Taylor & Francis Journals, vol. 32(3), pages 384-414, November.
    17. Martin Huber, 2010. "Identification of average treatment effects in social experiments under different forms of attrition," University of St. Gallen Department of Economics working paper series 2010 2010-22, Department of Economics, University of St. Gallen.
    18. Abdullah Kumas & Daniel L. Millimet, 2018. "Reassessing the effects of bilateral tax treaties on US FDI activity," Journal of Economics and Finance, Springer;Academy of Economics and Finance, vol. 42(3), pages 451-470, July.
    19. Cairo, Sofie & Mahlstedt, Robert, 2023. "The disparate effects of information provision: A field experiment on the work incentives of social welfare," Journal of Public Economics, Elsevier, vol. 226(C).
    20. Balestra, Simone & Backes-Gellner, Uschi, 2017. "Heterogeneous returns to education over the wage distribution: Who profits the most?," Labour Economics, Elsevier, vol. 44(C), pages 89-105.
    21. Shunyuan Zhang & Nitin Mehta & Param Vir Singh & Kannan Srinivasan, 2021. "Frontiers: Can an Artificial Intelligence Algorithm Mitigate Racial Economic Inequality? An Analysis in the Context of Airbnb," Marketing Science, INFORMS, vol. 40(5), pages 813-820, September.

    More about this item

    Keywords

    income support; methodological development;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:36:y:2012:i:6:p:449-474. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.