IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v42y2018i5-6p550-574.html
   My bibliography  Save this article

The Importance of Using Multiple Data Sources in Policy Assessments: Lessons From Two Conditional Cash Transfer Programs in New York City

Author

Listed:
  • Edith Yang
  • Richard Hendra

Abstract

Background: The high costs of implementing surveys are increasingly leading research teams to either cut back on surveys or to rely on administrative records. Yet no policy should be based on a single set of estimates, and every approach has its weaknesses. A mixture of approaches, each with its own biases, should provide the analyst with a better understanding of the underlying phenomenon. This claim is illustrated with a comparison of employment effect estimates of two conditional cash transfer programs in New York City using survey and administrative unemployment insurance (UI) data. Objectives: This article explores whether using administrative data and survey data produce different impact estimates and investigates the source of differential effects between data sources. Research design: The results of a survey nonresponse bias analysis and an analysis of characteristics of non-UI-covered job characteristics using data collected on 6,000 families who enrolled in either the Family Rewards or Work Rewards evaluation are presented. Results: In both evaluations, survey data showed positive employment effects, while administrative data showed no statistically significant employment effects. Family Rewards increased employment mostly in non-UI-covered jobs, while the positive survey impact estimates in Work Rewards were partially due to survey nonresponse bias. Conclusions: Despite cost pressures leading researchers to collect and analyze only administrative records, the results suggest that survey and administrative records data both suffer from different kinds of sample attrition, and researchers may need to triangulate data sources to draw accurate conclusions about program effects. Developing more economical data collection practices is a major priority.

Suggested Citation

  • Edith Yang & Richard Hendra, 2018. "The Importance of Using Multiple Data Sources in Policy Assessments: Lessons From Two Conditional Cash Transfer Programs in New York City," Evaluation Review, , vol. 42(5-6), pages 550-574, October.
    • Handle: RePEc:sae:evarev:v:42:y:2018:i:5-6:p:550-574
    DOI: 10.1177/0193841X18799820
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X18799820
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X18799820?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Katharine G. Abraham & John Haltiwanger & Kristin Sandusky & James R. Spletzer, 2013. "Exploring Differences in Employment between Household and Establishment Data," Journal of Labor Economics, University of Chicago Press, vol. 31(S1), pages 129-172.
    2. Reuben Ford & Douwêrê Grékou & Isaac Kwakye & Taylor Shek-wai Hui, 2018. "The Sensitivity of Impact Estimates to Data Sources Used: Analysis From an Access to Postsecondary Education Experiment," Evaluation Review, , vol. 42(5-6), pages 575-615, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Burt S. Barnow & David H. Greenberg, 2019. "Special Issue Editors’ Essay," Evaluation Review, , vol. 43(5), pages 231-265, October.
    2. Dang, Hai-Anh H. & Serajuddin, Umar, 2020. "Tracking the sustainable development goals: Emerging measurement challenges and further reflections," World Development, Elsevier, vol. 127(C).
    3. Bruce D. Meyer & Derek Wu & Victoria R. Mooers & Carla Medalia, 2019. "The use and misuse of income data and extreme poverty in the United States," AEI Economics Working Papers 1018925, American Enterprise Institute.
    4. Dang, Hai-Anh & Carletto, Calogero, 2022. "Recall Bias Revisited: Measure Farm Labor Using Mixed-Mode Surveys and Multiple Imputation," IZA Discussion Papers 14997, Institute of Labor Economics (IZA).
    5. Chinhui Juhn & Kristin McCue, 2012. "Workplace Characteristics and Employment of Older Workers," Working Papers 12-31, Center for Economic Studies, U.S. Census Bureau.
    6. Fredrik Andersson & Mónica García-Pérez & John Haltiwanger & Kristin McCue & Seth Sanders, 2014. "Workplace Concentration of Immigrants," Demography, Springer;Population Association of America (PAA), vol. 51(6), pages 2281-2306, December.
    7. Quinn Moore & Irma Perez-Johnson & Robert Santillano, 2018. "Decomposing Differences in Impacts on Survey- and Administrative-Measured Earnings From a Job Training Voucher Experiment," Evaluation Review, , vol. 42(5-6), pages 515-549, October.
    8. Hirsch, Barry & Macpherson, David A. & Qu, Zichong (Tom), 2022. "Real Estate Agent Earnings and Local Housing Prices," IZA Discussion Papers 15300, Institute of Labor Economics (IZA).
    9. Robert Moffitt & Sisi Zhang, 2022. "Estimating Trends in Male Earnings Volatility with the Panel Study of Income Dynamics," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 41(1), pages 20-25, December.
    10. Sarah Xue Dong, 2016. "Consistency between Sakernas and the IFLS for Analyses of Indonesia’s Labour Market: A Cross-Validation Exercise," Bulletin of Indonesian Economic Studies, Taylor & Francis Journals, vol. 52(3), pages 343-378, September.
    11. Aaron Flaaen & Matthew D. Shapiro & Isaac Sorkin, 2019. "Reconsidering the Consequences of Worker Displacements: Firm versus Worker Perspective," American Economic Journal: Macroeconomics, American Economic Association, vol. 11(2), pages 193-227, April.
    12. Jonathan Meer & Jeremy West, 2016. "Effects of the Minimum Wage on Employment Dynamics," Journal of Human Resources, University of Wisconsin Press, vol. 51(2), pages 500-522.
    13. Robert Moffitt & Sisi Zhang, 2018. "Income Volatility and the PSID: Past Research and New Results," AEA Papers and Proceedings, American Economic Association, vol. 108, pages 277-280, May.
    14. Keith A. Bailey & James R. Spletzer, 2020. "A New Measure of Multiple Jobholding in the U.S. Economy," Working Papers 20-26, Center for Economic Studies, U.S. Census Bureau.
    15. Liu, Ding & Millimet, Daniel L., 2020. "Bounding the Joint Distribution of Disability and Employment with Contaminated Data," IZA Discussion Papers 13020, Institute of Labor Economics (IZA).
    16. Katharine G. Abraham & John C. Haltiwanger & Kristin Sandusky & James R. Spletzer, 2017. "Measuring the Gig Economy: Current Knowledge and Open Issues," NBER Chapters, in: Measuring and Accounting for Innovation in the Twenty-First Century, pages 257-298, National Bureau of Economic Research, Inc.
    17. Dmitri Koustas, 2020. "Insights from New Tax-Based Measures of Gig Work in the United States," CESifo Forum, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 21(03), pages 5-9, September.
    18. Michael D. Carr & Robert A. Moffitt & Emily E. Wiemers, 2020. "Reconciling Trends in Volatility: Evidence from the SIPP Survey and Administrative Data," NBER Working Papers 27672, National Bureau of Economic Research, Inc.
    19. Kevin F. Hallock, 2013. "Data Improvement and Labor Economics," Journal of Labor Economics, University of Chicago Press, vol. 31(S1), pages 1-16.
    20. Celik Sule & Juhn Chinhui & McCue Kristin & Thompson Jesse, 2012. "Recent Trends in Earnings Volatility: Evidence from Survey and Administrative Data," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 12(2), pages 1-26, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:42:y:2018:i:5-6:p:550-574. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.