IDEAS home Printed from https://ideas.repec.org/a/eee/econom/v238y2024i2s030440762300297x.html
   My bibliography  Save this article

What leads to measurement errors? Evidence from reports of program participation in three surveys

Author

Listed:
  • Celhay, Pablo
  • Meyer, Bruce D.
  • Mittag, Nikolas

Abstract

Measurement errors are often a large source of bias in survey data. Lack of knowledge of the determinants of such errors makes it difficult to reduce the extent of errors when collecting data and to assess the validity of analyses using the data. We study the determinants of reporting error using high quality administrative data on government transfers linked to three major U.S. surveys. Our results support several theories of misreporting: Errors are related to event recall, forward and backward telescoping, salience of receipt, the stigma of reporting participation in welfare programs and respondent's degree of cooperation with the survey overall. We provide evidence on how survey design choices affect reporting errors. Our findings help survey users to gauge the reliability of their data and to devise estimation strategies that can correct for systematic errors, such as instrumental variable approaches. Understanding survey errors allows researchers collecting survey data to reduce them by improving survey design. Our results indicate that survey design should take into account that higher response rates as well as collecting more detailed information may have negative effects on survey accuracy.

Suggested Citation

  • Celhay, Pablo & Meyer, Bruce D. & Mittag, Nikolas, 2024. "What leads to measurement errors? Evidence from reports of program participation in three surveys," Journal of Econometrics, Elsevier, vol. 238(2).
  • Handle: RePEc:eee:econom:v:238:y:2024:i:2:s030440762300297x
    DOI: 10.1016/j.jeconom.2023.105581
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S030440762300297X
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.jeconom.2023.105581?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Paul Niehaus & Sandip Sukhtankar, 2013. "Corruption Dynamics: The Golden Goose Effect," American Economic Journal: Economic Policy, American Economic Association, vol. 5(4), pages 230-269, November.
    2. Jeffrey E. Zabel, 1998. "An Analysis of Attrition in the Panel Study of Income Dynamics and the Survey of Income and Program Participation with an Application to a Model of Labor Market Behavior," Journal of Human Resources, University of Wisconsin Press, vol. 33(2), pages 479-506.
    3. Bruce D. Meyer & Nikolas Mittag, 2019. "Using Linked Survey and Administrative Data to Better Measure Income: Implications for Poverty, Program Effectiveness, and Holes in the Safety Net," American Economic Journal: Applied Economics, American Economic Association, vol. 11(2), pages 176-204, April.
    4. Christopher Blattman & Julian C. Jamison & Margaret Sheridan, 2017. "Reducing Crime and Violence: Experimental Evidence from Cognitive Behavioral Therapy in Liberia," American Economic Review, American Economic Association, vol. 107(4), pages 1165-1206, April.
    5. Fraker, Thomas & Moffitt, Robert, 1988. "The effect of food stamps on labor supply : A bivariate selection model," Journal of Public Economics, Elsevier, vol. 35(1), pages 25-56, February.
    6. Bruce D. Meyer & Wallace K. C. Mok & James X. Sullivan, 2015. "Household Surveys in Crisis," Journal of Economic Perspectives, American Economic Association, vol. 29(4), pages 199-226, Fall.
    7. Karlan, Dean S. & Zinman, Jonathan, 2012. "List randomization for sensitive behavior: An application for measuring use of loan proceeds," Journal of Development Economics, Elsevier, vol. 98(1), pages 71-75.
    8. Ridder, Geert & Moffitt, Robert, 2007. "The Econometrics of Data Combination," Handbook of Econometrics, in: J.J. Heckman & E.E. Leamer (ed.), Handbook of Econometrics, edition 1, volume 6, chapter 75, Elsevier.
    9. Hoynes, Hilary Williamson & Schanzenbach, Diane Whitmore, 2012. "Work incentives and the Food Stamp Program," Journal of Public Economics, Elsevier, vol. 96(1), pages 151-162.
    10. Butler, J S, et al, 1987. "Measurement Error in Self-reported Health Variables," The Review of Economics and Statistics, MIT Press, vol. 69(4), pages 644-650, November.
    11. Celhay, Pablo & Meyer, Bruce D. & Mittag, Nikolas, 2021. "Errors in Reporting and Imputation of Government Benefits and Their Implications," IZA Discussion Papers 14396, Institute of Labor Economics (IZA).
    12. Charles Courtemanche & Augustine Denteh & Rusty Tchernis, 2019. "Estimating the Associations between SNAP and Food Insecurity, Obesity, and Food Purchases with Imperfect Administrative Measures of Participation," Southern Economic Journal, John Wiley & Sons, vol. 86(1), pages 202-228, July.
    13. Philip Gleason & Peter Schochet & Robert Moffitt, 1998. "The Dynamics of Food Stamp Program Participation in the Early 1990s," Mathematica Policy Research Reports ab95304cd2204323a950b50dd, Mathematica Policy Research.
    14. repec:mpr:mprres:1855 is not listed on IDEAS
    15. Robert A. Moffitt, 2016. "Economics of Means-Tested Transfer Programs in the United States, Volume 2," NBER Books, National Bureau of Economic Research, Inc, number moff-3.
    16. Celhay, Pablo & Meyer, Bruce D. & Mittag, Nikolas, 2022. "Stigma in Welfare Programs," IZA Discussion Papers 15431, Institute of Labor Economics (IZA).
    17. Craig Gundersen & Victor Oliveira, 2001. "The Food Stamp Program and Food Insufficiency," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 83(4), pages 875-887.
    18. Robert A. Moffitt, 2016. "Economics of Means-Tested Transfer Programs in the United States, Volume 1," NBER Books, National Bureau of Economic Research, Inc, number moff14-1.
    19. Ann Cartwright, 1957. "The Effect of Obtaining Information from Different Informants on a Family Morbidity Inquiry," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 6(1), pages 18-25, March.
    20. Besley, Timothy & Coate, Stephen, 1992. "Understanding welfare stigma: Taxpayer resentment and statistical discrimination," Journal of Public Economics, Elsevier, vol. 48(2), pages 165-183, July.
    21. Moffitt, Robert, 1983. "An Economic Model of Welfare Stigma," American Economic Review, American Economic Association, vol. 73(5), pages 1023-1035, December.
    22. Kerstin Bruckmeier & Gerrit M�ller & Regina T. Riphahn, 2014. "Who misreports welfare receipt in surveys?," Applied Economics Letters, Taylor & Francis Journals, vol. 21(12), pages 812-816, August.
    23. Rebecca M. Blank & Patricia Ruggles, 1996. "When Do Women Use Aid to Families with Dependent Children and Food Stamps? The Dynamics of Eligibility Versus Participation," Journal of Human Resources, University of Wisconsin Press, vol. 31(1), pages 57-89.
    24. Meyer, Bruce D. & Mittag, Nikolas, 2021. "An empirical total survey error decomposition using data combination," Journal of Econometrics, Elsevier, vol. 224(2), pages 286-305.
    25. Bollinger, Christopher R & David, Martin H, 2001. "Estimation with Response Error and Nonresponse: Food-Stamp Participation in the SIPP," Journal of Business & Economic Statistics, American Statistical Association, vol. 19(2), pages 129-141, April.
    26. Jeffrey Grogger, 2002. "The Behavioral Effects of Welfare Time Limits," American Economic Review, American Economic Association, vol. 92(2), pages 385-389, May.
    27. Caroline Danielson & Jacob Alex Klerman, 2006. "Why Did the Food Stamp Caseload Decline (and Rise)? Effects of Policies on the Economy," Working Papers 386, RAND Corporation.
    28. Gregory Acs & Katherin Ross Phillips & Sandi Nelson, 2005. "The Road Not Taken? Changes in Welfare Entry During the 1990s," Social Science Quarterly, Southwestern Social Science Association, vol. 86(s1), pages 1060-1079, December.
    29. Deborah Wagner & Mary Lane, 2014. "The Person Identification Validation System (PVS): Applying the Center for Administrative Records Research and Applications’ (CARRA) Record Linkage Software," CARRA Working Papers 2014-01, Center for Economic Studies, U.S. Census Bureau.
    30. Bruce D. Meyer & Derek Wu, 2018. "The Poverty Reduction of Social Security and Means-Tested Transfers," NBER Working Papers 24567, National Bureau of Economic Research, Inc.
    31. John Gibson, 2002. "Why Does the Engel Method Work? Food Demand, Economies of Size and Household Survey Methods," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 64(4), pages 341-359, September.
    32. Wooldridge, Jeffrey M., 2007. "Inverse probability weighted estimation for general missing data problems," Journal of Econometrics, Elsevier, vol. 141(2), pages 1281-1301, December.
    33. Dean Karlan & Jonathan Zinman, 2008. "Lying About Borrowing," Journal of the European Economic Association, MIT Press, vol. 6(2-3), pages 510-521, 04-05.
    34. Bruckmeier, Kerstin & Müller, Gerrit & Riphahn, Regina T., 2015. "Survey misreporting of welfare receipt—Respondent, interviewer, and interview characteristics," Economics Letters, Elsevier, vol. 129(C), pages 103-107.
    35. Bruce D. Meyer & Nikolas Mittag, 2019. "Misreporting of Government Transfers: How Important Are Survey Design and Geography?," Southern Economic Journal, John Wiley & Sons, vol. 86(1), pages 230-253, July.
    36. Bruce D. Meyer & Derek Wu, 2018. "The Poverty Reduction of Social Security and Means-Tested Transfers," ILR Review, Cornell University, ILR School, vol. 71(5), pages 1106-1153, October.
    37. Caroline Danielson & Jacob Alex Klerman, 2006. "Why Did the Food Stamp Caseload Decline (and Rise)? Effects of Policies on the Economy," Working Papers WR-386, RAND Corporation.
    38. Douglas S. Massey & Roger Tourangeau, 2013. "Where Do We Go from Here? Nonresponse and Social Measurement," The ANNALS of the American Academy of Political and Social Science, , vol. 645(1), pages 222-236, January.
    39. Nikolas Mittag, 2019. "Correcting for Misreporting of Government Benefits," American Economic Journal: Economic Policy, American Economic Association, vol. 11(2), pages 142-164, May.
    40. Moffitt, Robert A. (ed.), 2016. "Economics of Means-Tested Transfer Programs in the United States, Volume II," National Bureau of Economic Research Books, University of Chicago Press, number 9780226392493.
    41. David C. Ribar, 2005. "Transitions from Welfare and the Employment Prospects of Low‐Skill Workers," Southern Economic Journal, John Wiley & Sons, vol. 71(3), pages 514-533, January.
    42. Black, Dan & Sanders, Seth & Taylor, Lowell, 2003. "Measurement of Higher Education in the Census and Current Population Survey," Journal of the American Statistical Association, American Statistical Association, vol. 98, pages 545-554, January.
    43. Moffitt, Robert A. (ed.), 2016. "Economics of Means-Tested Transfer Programs in the United States, Volume I," National Bureau of Economic Research Books, University of Chicago Press, number 9780226370477.
    44. Bruce D. Meyer & Nikolas Mittag & Robert M. Goerge, 2022. "Errors in Survey Reporting and Imputation and Their Effects on Estimates of Food Stamp Program Participation," Journal of Human Resources, University of Wisconsin Press, vol. 57(5), pages 1605-1644.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Evan S. Totty & Thor Watson, 2024. "Privacy Protection and Accuracy: What Do We Know? Do We Know Things?? Let's Find Out!," NBER Chapters, in: Data Privacy Protection and the Conduct of Applied Research: Methods, Approaches and their Consequences, National Bureau of Economic Research, Inc.
    2. Adam Bee & Irena Dushi & Joshua Mitchell & Brad Trenkamp, 2024. "Measuring Income of the Aged in Household Surveys: Evidence from Linked Administrative Records," Working Papers 24-32, Center for Economic Studies, U.S. Census Bureau.
    3. Li, Haizheng & Liu, Qinyi & Xu, Yiting, 2024. "Noncognitive Human Capital and Misreporting Behavior in Online Surveys," IZA Discussion Papers 17332, Institute of Labor Economics (IZA).
    4. Richiardi, Matteo & Vella, Melchior, 2024. "Mind vs matter: economic and psychologic determinants of take-up rates of social benefits in the UK," Centre for Microsimulation and Policy Analysis Working Paper Series CEMPA6/24, Centre for Microsimulation and Policy Analysis at the Institute for Social and Economic Research.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Celhay, Pablo & Meyer, Bruce D. & Mittag, Nikolas, 2022. "What Leads to Measurement Errors? Evidence from Reports of Program Participation in Three Surveys," IZA Discussion Papers 14995, Institute of Labor Economics (IZA).
    2. Bruce D. Meyer & Nikolas Mittag & Derek Wu, 2024. "Race, Ethnicity, and Measurement Error," NBER Chapters, in: Race, Ethnicity, and Economic Statistics for the 21st Century, National Bureau of Economic Research, Inc.
    3. Meyer, Bruce D. & Mittag, Nikolas, 2021. "An empirical total survey error decomposition using data combination," Journal of Econometrics, Elsevier, vol. 224(2), pages 286-305.
    4. Ha Trong Nguyen & Huong Thu Le & Luke Connelly & Francis Mitrou, 2023. "Accuracy of self‐reported private health insurance coverage," Health Economics, John Wiley & Sons, Ltd., vol. 32(12), pages 2709-2729, December.
    5. Bruce D. Meyer & Nikolas Mittag, 2019. "Combining Administrative and Survey Data to Improve Income Measurement," NBER Working Papers 25738, National Bureau of Economic Research, Inc.
    6. Celhay, Pablo & Meyer, Bruce D. & Mittag, Nikolas, 2022. "Stigma in Welfare Programs," IZA Discussion Papers 15431, Institute of Labor Economics (IZA).
    7. Meyer, Bruce D. & Mittag, Nikolas, 2019. "Combining Administrative and Survey Data to Improve Income Measurement," IZA Discussion Papers 12266, Institute of Labor Economics (IZA).
    8. Bruce D. Meyer & Derek Wu & Victoria R. Mooers & Carla Medalia, 2019. "The use and misuse of income data and extreme poverty in the United States," AEI Economics Working Papers 1018925, American Enterprise Institute.
    9. Bruckmeier, Kerstin & Riphahn, Regina T. & Wiemers, Jürgen, 2019. "Benefit underreporting in survey data and its consequences for measuring non-take-up: new evidence from linked administrative and survey data," IAB-Discussion Paper 201906, Institut für Arbeitsmarkt- und Berufsforschung (IAB), Nürnberg [Institute for Employment Research, Nuremberg, Germany].
    10. Ha Trong Nguyen & Huong Thu Le & Luke B Connelly, 2021. "Who's declining the “free lunch”? New evidence from the uptake of public child dental benefits," Health Economics, John Wiley & Sons, Ltd., vol. 30(2), pages 270-288, February.
    11. Meyer, Bruce D. & Mittag, Nikolas & Wu, Derek, 2024. "Race, Ethnicity, and Measurement Error," IZA Discussion Papers 17349, Institute of Labor Economics (IZA).
    12. Adam Bee & Joshua Mitchell & Nikolas Mittag & Jonathan Rothbaum & Carl Sanders & Lawrence Schmidt & Matthew Unrath, 2023. "National Experimental Wellbeing Statistics - Version 1," Working Papers 23-04, Center for Economic Studies, U.S. Census Bureau.
    13. Bruce D. Meyer & Derek Wu, 2018. "The Poverty Reduction of Social Security and Means-Tested Transfers," NBER Working Papers 24567, National Bureau of Economic Research, Inc.
    14. Melanie Guldi & Lucie Schmidt, 2017. "Taxes, Transfers, and Women’s Labor Supply in the United States," Working Papers 2017-01, University of Central Florida, Department of Economics.
    15. Huffman, Sonya Kostova & Jensen, Helen H., 2003. "Do Food Assistance Programs Improve Household Food Security?: Recent Evidence From The United States," 2003 Annual meeting, July 27-30, Montreal, Canada 22219, American Agricultural Economics Association (New Name 2008: Agricultural and Applied Economics Association).
    16. Craig Gundersen & Brent Kreider, 2008. "Food Stamps and Food Insecurity: What Can Be Learned in the Presence of Nonclassical Measurement Error?," Journal of Human Resources, University of Wisconsin Press, vol. 43(2), pages 352-382.
    17. Gizem Kosar & Robert A. Moffitt, 2017. "Trends in Cumulative Marginal Tax Rates Facing Low-Income Families, 1997-2007," Tax Policy and the Economy, University of Chicago Press, vol. 31(1), pages 43-70.
    18. Masayoshi Hayashi, 2021. "Transfer Benefits, Implicit Taxes, and the Earnings of Welfare Recipients: Evidence from Public Assistance Programs in Japan," CIRJE F-Series CIRJE-F-1164, CIRJE, Faculty of Economics, University of Tokyo.
    19. Marina Gindelsky, 2022. "Do transfers lower inequality between households? Demographic evidence from Distributional National Accounts," Economic Inquiry, Western Economic Association International, vol. 60(3), pages 1233-1257, July.
    20. Helen H. Jensen & Brent Kreider & Oleksandr Zhylyevskyy, 2019. "Investigating Treatment Effects of Participating Jointly in SNAP and WIC when the Treatment Is Validated Only for SNAP," Southern Economic Journal, John Wiley & Sons, vol. 86(1), pages 124-155, July.

    More about this item

    Keywords

    Measurement error; Welfare programs; Survey methods; Validation; Misreporting;
    All these keywords.

    JEL classification:

    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • H53 - Public Economics - - National Government Expenditures and Related Policies - - - Government Expenditures and Welfare Programs
    • I3 - Health, Education, and Welfare - - Welfare, Well-Being, and Poverty

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:econom:v:238:y:2024:i:2:s030440762300297x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/jeconom .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.