IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/29652.html
   My bibliography  Save this paper

What Leads to Measurement Errors? Evidence from Reports of Program Participation in Three Surveys

Author

Listed:
  • Pablo A. Celhay
  • Bruce D. Meyer
  • Nikolas Mittag

Abstract

Measurement errors are often a large source of bias in survey data. Lack of knowledge of the determinants of such errors makes it difficult for data producers to reduce the extent of errors and for data users to assess the validity of analyses using the data. We study the determinants of reporting error using high quality administrative data on government transfers linked to three major U.S. surveys. Our results support several theories of misreporting: Errors are related to event recall, forward and backward telescoping, salience of receipt, the stigma of reporting participation in welfare programs and respondent’s degree of cooperation with the survey overall. We provide evidence on how survey design choices affect reporting errors. Our findings help survey users to gauge the reliability of their data and to devise estimation strategies that can correct for systematic errors, such as instrumental variable approaches. Understanding survey errors allows survey producers to reduce them by improving survey design. Our results indicate that survey producers should take into account that higher response rates as well as collecting more detailed information may have negative effects on survey accuracy.

Suggested Citation

  • Pablo A. Celhay & Bruce D. Meyer & Nikolas Mittag, 2022. "What Leads to Measurement Errors? Evidence from Reports of Program Participation in Three Surveys," NBER Working Papers 29652, National Bureau of Economic Research, Inc.
    • Handle: RePEc:nbr:nberwo:29652
    Note: CH LS PE
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w29652.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Bruce D. Meyer & Nikolas Mittag, 2015. "Using Linked Survey and Administrative Data to Better Measure Income: Implications for Poverty, Program Effectiveness and Holes in the Safety Net," Upjohn Working Papers 15-242, W.E. Upjohn Institute for Employment Research.
    2. Nguimkeu, Pierre & Denteh, Augustine & Tchernis, Rusty, 2019. "On the estimation of treatment effects with endogenous misreporting," Journal of Econometrics, Elsevier, vol. 208(2), pages 487-506.
    3. Moffitt, Robert, 1983. "An Economic Model of Welfare Stigma," American Economic Review, American Economic Association, vol. 73(5), pages 1023-1035, December.
    4. Bruce D. Meyer & Derek Wu, 2018. "The Poverty Reduction of Social Security and Means-Tested Transfers," NBER Working Papers 24567, National Bureau of Economic Research, Inc.
    5. Wooldridge, Jeffrey M., 2007. "Inverse probability weighted estimation for general missing data problems," Journal of Econometrics, Elsevier, vol. 141(2), pages 1281-1301, December.
    6. Karlan, Dean S. & Zinman, Jonathan, 2012. "List randomization for sensitive behavior: An application for measuring use of loan proceeds," Journal of Development Economics, Elsevier, vol. 98(1), pages 71-75.
    7. Bruce D. Meyer & Wallace K. C. Mok & James X. Sullivan, 2015. "Household Surveys in Crisis," Journal of Economic Perspectives, American Economic Association, vol. 29(4), pages 199-226, Fall.
    8. Bruce D. Meyer & Derek Wu, 2018. "The Poverty Reduction of Social Security and Means-Tested Transfers," ILR Review, Cornell University, ILR School, vol. 71(5), pages 1106-1153, October.
    9. Nikolas Mittag, 2019. "Correcting for Misreporting of Government Benefits," American Economic Journal: Economic Policy, American Economic Association, vol. 11(2), pages 142-164, May.
    10. Paul Niehaus & Sandip Sukhtankar, 2013. "Corruption Dynamics: The Golden Goose Effect," American Economic Journal: Economic Policy, American Economic Association, vol. 5(4), pages 230-269, November.
    11. Jeffrey E. Zabel, 1998. "An Analysis of Attrition in the Panel Study of Income Dynamics and the Survey of Income and Program Participation with an Application to a Model of Labor Market Behavior," Journal of Human Resources, University of Wisconsin Press, vol. 33(2), pages 479-506.
    12. Dean Karlan & Jonathan Zinman, 2008. "Lying About Borrowing," Journal of the European Economic Association, MIT Press, vol. 6(2-3), pages 510-521, 04-05.
    13. Bruce D. Meyer & Nikolas Mittag, 2019. "Misreporting of Government Transfers: How Important Are Survey Design and Geography?," Southern Economic Journal, John Wiley & Sons, vol. 86(1), pages 230-253, July.
    14. Meyer, Bruce D. & Mittag, Nikolas, 2021. "An empirical total survey error decomposition using data combination," Journal of Econometrics, Elsevier, vol. 224(2), pages 286-305.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Nicolas Frémeaux, 2023. "The More, the Better? Individual and Joint Interviewing in Surveys," Annals of Economics and Statistics, GENES, issue 149, pages 63-96.
    2. Ha Trong Nguyen & Huong Thu Le & Luke Connelly & Francis Mitrou, 2023. "Accuracy of self‐reported private health insurance coverage," Health Economics, John Wiley & Sons, Ltd., vol. 32(12), pages 2709-2729, December.
    3. Celhay, Pablo & Meyer, Bruce D. & Mittag, Nikolas, 2022. "Stigma in Welfare Programs," IZA Discussion Papers 15431, Institute of Labor Economics (IZA).
    4. Cameron Deal & Shea Greenberg & Gilbert Gonzales, 2024. "Sexual identity, poverty, and utilization of government services," Journal of Population Economics, Springer;European Society for Population Economics, vol. 37(2), pages 1-31, June.
    5. Krista Ruffini, 2023. "Does Unconditional Cash during Pregnancy Affect Infant Health?," Opportunity and Inclusive Growth Institute Working Papers 072, Federal Reserve Bank of Minneapolis.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Celhay, Pablo & Meyer, Bruce D. & Mittag, Nikolas, 2024. "What leads to measurement errors? Evidence from reports of program participation in three surveys," Journal of Econometrics, Elsevier, vol. 238(2).
    2. Bruce D. Meyer & Derek Wu & Victoria R. Mooers & Carla Medalia, 2019. "The Use and Misuse of Income Data and Extreme Poverty in the United States," NBER Working Papers 25907, National Bureau of Economic Research, Inc.
    3. Adam Bee & Joshua Mitchell & Nikolas Mittag & Jonathan Rothbaum & Carl Sanders & Lawrence Schmidt & Matthew Unrath, 2023. "National Experimental Wellbeing Statistics - Version 1," Working Papers 23-04, Center for Economic Studies, U.S. Census Bureau.
    4. Bruce D. Meyer & Nikolas Mittag, 2019. "Combining Administrative and Survey Data to Improve Income Measurement," NBER Working Papers 25738, National Bureau of Economic Research, Inc.
    5. Meyer, Bruce D. & Mittag, Nikolas, 2019. "Combining Administrative and Survey Data to Improve Income Measurement," IZA Discussion Papers 12266, Institute of Labor Economics (IZA).
    6. Gabrielle Pepin, 2022. "The effects of welfare time limits on access to financial resources: Evidence from the 2010s," Southern Economic Journal, John Wiley & Sons, vol. 88(4), pages 1343-1372, April.
    7. Ha Trong Nguyen & Huong Thu Le & Luke Connelly & Francis Mitrou, 2023. "Accuracy of self‐reported private health insurance coverage," Health Economics, John Wiley & Sons, Ltd., vol. 32(12), pages 2709-2729, December.
    8. Nikolas Mittag, 2019. "Correcting for Misreporting of Government Benefits," American Economic Journal: Economic Policy, American Economic Association, vol. 11(2), pages 142-164, May.
    9. James X. Sullivan, 2020. "A Cautionary Tale of Using Data From the Tail," Demography, Springer;Population Association of America (PAA), vol. 57(6), pages 2361-2368, December.
    10. Parolin, Zachary & Brady, David, 2018. "Extreme Child Poverty and the Role of Social Policy in the United States," SocArXiv u5ecn, Center for Open Science.
    11. Mittag, Nikolas, 2016. "Correcting for Misreporting of Government Benefits," IZA Discussion Papers 10266, Institute of Labor Economics (IZA).
    12. Kyung Min Kang & Robert A. Moffitt, 2019. "The Effect of SNAP and School Food Programs on Food Security, Diet Quality, and Food Spending: Sensitivity to Program Reporting Error," Southern Economic Journal, John Wiley & Sons, vol. 86(1), pages 156-201, July.
    13. Bruckmeier, Kerstin & Riphahn, Regina T. & Wiemers, Jürgen, 2019. "Benefit underreporting in survey data and its consequences for measuring non-take-up: new evidence from linked administrative and survey data," IAB-Discussion Paper 201906, Institut für Arbeitsmarkt- und Berufsforschung (IAB), Nürnberg [Institute for Employment Research, Nuremberg, Germany].
    14. Marina Gindelsky, 2022. "Do transfers lower inequality between households? Demographic evidence from Distributional National Accounts," Economic Inquiry, Western Economic Association International, vol. 60(3), pages 1233-1257, July.
    15. Zachary Parolin, 2019. "The Effect of Benefit Underreporting on Estimates of Poverty in the United States," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 144(2), pages 869-898, July.
    16. Korenman, Sanders & Remler, Dahlia K. & Hyson, Rosemary T., 2021. "Health insurance and poverty of the older population in the United States: The importance of a health inclusive poverty measure," The Journal of the Economics of Ageing, Elsevier, vol. 18(C).
    17. Luis Ayala & Ana Pérez & Mercedes Prieto-Alaiz, 2022. "The impact of different data sources on the level and structure of income inequality," SERIEs: Journal of the Spanish Economic Association, Springer;Spanish Economic Association, vol. 13(3), pages 583-611, September.
    18. Charles Courtemanche & Augustine Denteh & Rusty Tchernis, 2019. "Estimating the Associations between SNAP and Food Insecurity, Obesity, and Food Purchases with Imperfect Administrative Measures of Participation," Southern Economic Journal, John Wiley & Sons, vol. 86(1), pages 202-228, July.
    19. Bradley Hardy & Timothy Smeeding & James P. Ziliak, 2018. "The Changing Safety Net for Low-Income Parents and Their Children: Structural or Cyclical Changes in Income Support Policy?," Demography, Springer;Population Association of America (PAA), vol. 55(1), pages 189-221, February.
    20. Adrian Chadi, 2019. "Dissatisfied with life or with being interviewed? Happiness and the motivation to participate in a survey," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 53(3), pages 519-553, October.

    More about this item

    JEL classification:

    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • D31 - Microeconomics - - Distribution - - - Personal Income and Wealth Distribution
    • I32 - Health, Education, and Welfare - - Welfare, Well-Being, and Poverty - - - Measurement and Analysis of Poverty
    • I38 - Health, Education, and Welfare - - Welfare, Well-Being, and Poverty - - - Government Programs; Provision and Effects of Welfare Programs

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:29652. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.