IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/21447.html
   My bibliography  Save this paper

Measuring the Measurement Error: A Method to Qualitatively Validate Survey Data

Author

Listed:
  • Christopher Blattman
  • Julian C. Jamison
  • Tricia Koroknay-Palicz
  • Katherine Rodrigues
  • Margaret Sheridan

Abstract

Field experiments rely heavily on self-reported data, but subjects may misreport behaviors, especially sensitive ones such as crime. If treatment influences survey responses, it biases experimental estimates. We develop a validation technique that uses intensive qualitative work to assess survey measurement error. Subjects were assigned to receive cash, therapy, both, or neither. According to survey responses, receiving both treatments dramatically reduced crime and other sensitive behaviors. Local researchers spent several days with a random subsample of subjects following their endline surveys, building trust and seeking verbal confirmation of six behaviors: theft, drug use, homelessness, gambling, and two expenditures. This validation suggests that subjects in the control and cash only groups underreported sensitive behaviors and expenditures in the survey relative to the other treatment arms. We bound survey-based treatment effects estimates, and find the impacts of cash and therapy on crime may be larger than suggested by surveys alone.

Suggested Citation

  • Christopher Blattman & Julian C. Jamison & Tricia Koroknay-Palicz & Katherine Rodrigues & Margaret Sheridan, 2015. "Measuring the Measurement Error: A Method to Qualitatively Validate Survey Data," NBER Working Papers 21447, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:21447
    Note: DEV
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w21447.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. de Mel, Suresh & McKenzie, David J. & Woodruff, Christopher, 2009. "Measuring microenterprise profits: Must we ask how the sausage is made?," Journal of Development Economics, Elsevier, vol. 88(1), pages 19-31, January.
    2. Jerry Hausman, 2001. "Mismeasured Variables in Econometric Analysis: Problems from the Right and Problems from the Left," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 57-67, Fall.
    3. Jamison, Julian & Karlan, Dean S. & Raffler, Pia, 2013. "Mixed Method Evaluation of a Passive mHealth Sexual Information Testing Service in Uganda," Center Discussion Papers 150383, Yale University, Economic Growth Center.
    4. Beegle, Kathleen & De Weerdt, Joachim & Friedman, Jed & Gibson, John, 2012. "Methods of household consumption measurement through surveys: Experimental results from Tanzania," Journal of Development Economics, Elsevier, vol. 98(1), pages 3-18.
    5. Karlan, Dean S. & Zinman, Jonathan, 2012. "List randomization for sensitive behavior: An application for measuring use of loan proceeds," Journal of Development Economics, Elsevier, vol. 98(1), pages 71-75.
    6. Christopher Blattman & Julian C. Jamison & Margaret Sheridan, 2015. "Reducing crime and violence: Experimental evidence from cognitive behavioral therapy in Liberia," NBER Working Papers 21204, National Bureau of Economic Research, Inc.
    7. David J. Deming, 2011. "Better Schools, Less Crime?," The Quarterly Journal of Economics, Oxford University Press, vol. 126(4), pages 2063-2115.
    8. César Martinelli & Susan Wendy Parker, 2009. "Deception and Misreporting in a Social Program," Journal of the European Economic Association, MIT Press, vol. 7(4), pages 886-908, June.
    9. Jamison, Julian & Karlan, Dean & Raffler, Pia, 2013. "Mixed Method Evaluation of a Passive mHealth Sexual Information Texting Service in Uganda," Working Papers 116, Yale University, Department of Economics.
    10. Bound, John & Brown, Charles & Mathiowetz, Nancy, 2001. "Measurement error in survey data," Handbook of Econometrics,in: J.J. Heckman & E.E. Leamer (ed.), Handbook of Econometrics, edition 1, volume 5, chapter 59, pages 3705-3843 Elsevier.
    11. repec:cup:apsrev:v:103:y:2009:i:04:p:622-644_99 is not listed on IDEAS
    12. Bullock, Will & Imai, Kosuke & Shapiro, Jacob N., 2011. "Statistical Analysis of Endorsement Experiments: Measuring Support for Militant Groups in Pakistan," Political Analysis, Cambridge University Press, vol. 19(04), pages 363-384, September.
    13. Baird, Sarah & Özler, Berk, 2012. "Examining the reliability of self-reported data on school participation," Journal of Development Economics, Elsevier, vol. 98(1), pages 89-93.
    14. Dean Karlan & Jonathan Zinman, 2008. "Lying About Borrowing," Journal of the European Economic Association, MIT Press, vol. 6(2-3), pages 510-521, 04-05.
    15. Blair, Graeme & Imai, Kosuke, 2012. "Statistical Analysis of List Experiments," Political Analysis, Cambridge University Press, vol. 20(01), pages 47-77, December.
    16. McKenzie, David, 2012. "Beyond baseline and follow-up: The case for more T in experiments," Journal of Development Economics, Elsevier, vol. 99(2), pages 210-221.
    17. anonymous, 1997. "Fed amends consumer leasing act," Financial Update, Federal Reserve Bank of Atlanta, issue Oct, pages 1-5.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Christopher Blattman & Julian C. Jamison & Margaret Sheridan, 2017. "Reducing Crime and Violence: Experimental Evidence from Cognitive Behavioral Therapy in Liberia," American Economic Review, American Economic Association, vol. 107(4), pages 1165-1206, April.
    2. Bonan, Jacopo & Battiston, Pietro & Bleck, Jaimie & LeMay-Boucher, Philippe & Pareglio, Stefano & Sarr, Bassirou & Tavoni, Massimo, 2017. "Social Interaction and Technology Adoption: Experimental Evidence from Improved Cookstoves in Mali," MITP: Mitigation, Innovation and Transformation Pathways 263486, Fondazione Eni Enrico Mattei (FEEM).
    3. Christopher Blattman & Julian C. Jamison & Margaret Sheridan, 2015. "Reducing crime and violence: Experimental evidence from cognitive behavioral therapy in Liberia," NBER Working Papers 21204, National Bureau of Economic Research, Inc.
    4. Elisabetta de Cao & Clemens Lutz, 2015. "Measuring attitudes regarding female genital mutilation through a list experiment," CSAE Working Paper Series 2015-20, Centre for the Study of African Economies, University of Oxford.
    5. Ben Gillen & Erik Snowberg & Leeat Yariv, 2015. "Experimenting with Measurement Error: Techniques with Applications to the Caltech Cohort Study," NBER Working Papers 21517, National Bureau of Economic Research, Inc.
    6. van Bergeijk, P.A.G., 2017. "Measurement error of global production," ISS Working Papers - General Series 632, International Institute of Social Studies of Erasmus University Rotterdam (ISS), The Hague.

    More about this item

    JEL classification:

    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • I32 - Health, Education, and Welfare - - Welfare, Well-Being, and Poverty - - - Measurement and Analysis of Poverty
    • K4 - Law and Economics - - Legal Procedure, the Legal System, and Illegal Behavior
    • O1 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:21447. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (). General contact details of provider: http://edirc.repec.org/data/nberrus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.