IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/21517.html
   My bibliography  Save this paper

Experimenting with Measurement Error: Techniques with Applications to the Caltech Cohort Study

Author

Listed:
  • Ben Gillen
  • Erik Snowberg
  • Leeat Yariv

Abstract

Measurement error is ubiquitous in experimental work. It leads to imperfect statistical controls, attenuated estimated effects of elicited behaviors, and biased correlations between characteristics. We develop simple statistical techniques for dealing with experimental measurement error. These techniques are applied to data from the Caltech Cohort Study, which conducts repeated incentivized surveys of the Caltech student body. We illustrate the impact of measurement error by replicating three classic experiments, and showing that results change substantially when measurement error is taken into account. Collectively, these results show that failing to properly account for measurement error may cause a field-wide bias: it may lead scholars to identify "new" effects and phenomena that are actually similar to those previously documented.

Suggested Citation

  • Ben Gillen & Erik Snowberg & Leeat Yariv, 2015. "Experimenting with Measurement Error: Techniques with Applications to the Caltech Cohort Study," NBER Working Papers 21517, National Bureau of Economic Research, Inc.
    • Handle: RePEc:nbr:nberwo:21517
    Note: DEV LS POL
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w21517.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Alan B. Krueger, 2001. "Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 69-85, Fall.
    2. Catherine C. Eckel & Philip J. Grossman, 2002. "Sex Differences and Statistical Stereotyping in Attitudes Toward Financial Risk," Monash Economics Working Papers archive-03, Monash University, Department of Economics.
    3. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    4. Blattman, Christopher & Jamison, Julian & Koroknay-Palicz, Tricia & Rodrigues, Katherine & Sheridan, Margaret, 2016. "Measuring the measurement error: A method to qualitatively validate survey data," Journal of Development Economics, Elsevier, vol. 120(C), pages 99-112.
    5. Lisa Anderson & Jennifer Mellor, 2009. "Are risk preferences stable? Comparing an experimental measure with a validated survey-based measure," Journal of Risk and Uncertainty, Springer, vol. 39(2), pages 137-160, October.
    6. Thomas Dohmen & Armin Falk & David Huffman & Uwe Sunde & Jürgen Schupp & Gert G. Wagner, 2011. "Individual Risk Attitudes: Measurement, Determinants, And Behavioral Consequences," Journal of the European Economic Association, European Economic Association, vol. 9(3), pages 522-550, June.
    7. McKelvey Richard D. & Palfrey Thomas R., 1995. "Quantal Response Equilibria for Normal Form Games," Games and Economic Behavior, Elsevier, vol. 10(1), pages 6-38, July.
    8. Alexandre Belloni & Victor Chernozhukov & Christian Hansen, 2011. "Inference on Treatment Effects After Selection Amongst High-Dimensional Controls," Papers 1201.0224, arXiv.org, revised May 2012.
    9. Fan, Jianqing & Liao, Yuan, 2012. "Endogeneity in ultrahigh dimension," MPRA Paper 38698, University Library of Munich, Germany.
    10. Charles A. Holt & Susan K. Laury, 2002. "Risk Aversion and Incentive Effects," American Economic Review, American Economic Association, vol. 92(5), pages 1644-1655, December.
    11. Harless, David W & Camerer, Colin F, 1994. "The Predictive Utility of Generalized Expected Utility Theories," Econometrica, Econometric Society, vol. 62(6), pages 1251-1289, November.
    12. Hertz Tom & Jayasundera Tamara & Piraino Patrizio & Selcuk Sibel & Smith Nicole & Verashchagina Alina, 2008. "The Inheritance of Educational Inequality: International Comparisons and Fifty-Year Trends," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 7(2), pages 1-48, January.
    13. Blair Cleave & Nikos Nikiforakis & Robert Slonim, 2013. "Is there selection bias in laboratory experiments? The case of social and risk preferences," Experimental Economics, Springer;Economic Science Association, vol. 16(3), pages 372-382, September.
    14. Nagel, Rosemarie, 1995. "Unraveling in Guessing Games: An Experimental Study," American Economic Review, American Economic Association, vol. 85(5), pages 1313-1326, December.
    15. Segal, Uzi, 1987. "The Ellsberg Paradox and Risk Aversion: An Anticipated Utility Approach," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 28(1), pages 175-202, February.
    16. Fan J. & Li R., 2001. "Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties," Journal of the American Statistical Association, American Statistical Association, vol. 96, pages 1348-1360, December.
    17. G.M. Constantinides & M. Harris & R. M. Stulz (ed.), 2013. "Handbook of the Economics of Finance," Handbook of the Economics of Finance, Elsevier, volume 2, number 2-b.
    18. G.M. Constantinides & M. Harris & R. M. Stulz (ed.), 2013. "Handbook of the Economics of Finance," Handbook of the Economics of Finance, Elsevier, volume 2, number 2-a.
    19. Christoph Engel, 2011. "Dictator games: a meta study," Experimental Economics, Springer;Economic Science Association, vol. 14(4), pages 583-610, November.
    20. A. Belloni & V. Chernozhukov & L. Wang, 2011. "Square-root lasso: pivotal recovery of sparse signals via conic programming," Biometrika, Biometrika Trust, vol. 98(4), pages 791-806.
    21. Harrison, Glenn W. & Lau, Morten I. & Elisabet Rutström, E., 2009. "Risk attitudes, randomization to treatment, and self-selection into experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 70(3), pages 498-507, June.
    22. Yoram Halevy, 2007. "Ellsberg Revisited: An Experimental Study," Econometrica, Econometric Society, vol. 75(2), pages 503-536, March.
    23. Uri Gneezy & Jan Potters, 1997. "An Experiment on Risk Taking and Evaluation Periods," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 112(2), pages 631-645.
    24. Milton Friedman, 1957. "A Theory of the Consumption Function," NBER Books, National Bureau of Economic Research, Inc, number frie57-1.
    25. Andrews, Donald W K, 2001. "Testing When a Parameter Is on the Boundary of the Maintained Hypothesis," Econometrica, Econometric Society, vol. 69(3), pages 683-734, May.
    26. Sendhil Mullainathan & Marianne Bertrand, 2001. "Do People Mean What They Say? Implications for Subjective Survey Data," American Economic Review, American Economic Association, vol. 91(2), pages 67-72, May.
    27. Armin Falk & Stephan Meier & Christian Zehnder, 2013. "Do Lab Experiments Misrepresent Social Preferences? The Case Of Self-Selected Student Samples," Journal of the European Economic Association, European Economic Association, vol. 11(4), pages 839-852, August.
    28. Charness, Gary & Gneezy, Uri & Imas, Alex, 2013. "Experimental methods: Eliciting risk preferences," Journal of Economic Behavior & Organization, Elsevier, vol. 87(C), pages 43-51.
    29. Rachel Croson & Uri Gneezy, 2009. "Gender Differences in Preferences," Journal of Economic Literature, American Economic Association, vol. 47(2), pages 448-474, June.
    30. Chetan Dave & Catherine Eckel & Cathleen Johnson & Christian Rojas, 2010. "Eliciting risk preferences: When is simple better?," Journal of Risk and Uncertainty, Springer, vol. 41(3), pages 219-243, December.
    31. Liran Einav & Amy Finkelstein & Iuliana Pascu & Mark R. Cullen, 2012. "How General Are Risk Preferences? Choices under Uncertainty in Different Domains," American Economic Review, American Economic Association, vol. 102(6), pages 2606-2638, October.
    32. Jerry Hausman, 2001. "Mismeasured Variables in Econometric Analysis: Problems from the Right and Problems from the Left," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 57-67, Fall.
    33. Milton Friedman, 1957. "Introduction to "A Theory of the Consumption Function"," NBER Chapters, in: A Theory of the Consumption Function, pages 1-6, National Bureau of Economic Research, Inc.
    34. Bound, John & Brown, Charles & Mathiowetz, Nancy, 2001. "Measurement error in survey data," Handbook of Econometrics, in: J.J. Heckman & E.E. Leamer (ed.), Handbook of Econometrics, edition 1, volume 5, chapter 59, pages 3705-3843, Elsevier.
    35. Leeb, Hannes & Pötscher, Benedikt M., 2005. "Model Selection And Inference: Facts And Fiction," Econometric Theory, Cambridge University Press, vol. 21(1), pages 21-59, February.
    36. Eckel, Catherine C. & Grossman, Philip J., 2008. "Men, Women and Risk Aversion: Experimental Evidence," Handbook of Experimental Economics Results, in: Charles R. Plott & Vernon L. Smith (ed.), Handbook of Experimental Economics Results, edition 1, volume 1, chapter 113, pages 1061-1073, Elsevier.
    37. repec:fth:prinin:455 is not listed on IDEAS
    38. Barber, Brad M. & Odean, Terrance, 2013. "The Behavior of Individual Investors," Handbook of the Economics of Finance, in: G.M. Constantinides & M. Harris & R. M. Stulz (ed.), Handbook of the Economics of Finance, volume 2, chapter 0, pages 1533-1570, Elsevier.
    39. Shane Frederick, 2005. "Cognitive Reflection and Decision Making," Journal of Economic Perspectives, American Economic Association, vol. 19(4), pages 25-42, Fall.
    40. Joshua Angrist & Alan Krueger, 2001. "Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments," Working Papers 834, Princeton University, Department of Economics, Industrial Relations Section..
    41. Friedman, Daniel & Isaac, R. Mark & James, Duncan & Sunder, Shyam, 2014. "Risky Curves: On the Empirical Failure of Expected Utility," Santa Cruz Department of Economics, Working Paper Series qt87v8k86z, Department of Economics, UC Santa Cruz.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Paolo Crosetto & Antonio Filippin, 2016. "A theoretical and experimental appraisal of four risk elicitation methods," Experimental Economics, Springer;Economic Science Association, vol. 19(3), pages 613-641, September.
    2. Ranganathan, Kavitha & Lejarraga, Tomás, 2021. "Elicitation of risk preferences through satisficing," Journal of Behavioral and Experimental Finance, Elsevier, vol. 32(C).
    3. Galizzi, Matteo M. & Machado, Sara R. & Miniaci, Raffaele, 2016. "Temporal stability, cross-validity, and external validity of risk preferences measures: experimental evidence from a UK representative sample," LSE Research Online Documents on Economics 67554, London School of Economics and Political Science, LSE Library.
    4. Tamás Csermely & Alexander Rabas, 2016. "How to reveal people’s preferences: Comparing time consistency and predictive power of multiple price list risk elicitation methods," Journal of Risk and Uncertainty, Springer, vol. 53(2), pages 107-136, December.
    5. Strobl, Renate, 2022. "Background risk, insurance and investment behaviour: Experimental evidence from Kenya," Journal of Economic Behavior & Organization, Elsevier, vol. 202(C), pages 34-68.
    6. Menkhoff, Lukas & Sakha, Sahra, 2017. "Estimating risky behavior with multiple-item risk measures," Journal of Economic Psychology, Elsevier, vol. 59(C), pages 59-86.
    7. Antonio FILIPPIN & Paolo CROSETTO, 2014. "A Reconsideration of Gender Differences in Risk Attitudes," Departmental Working Papers 2014-01, Department of Economics, Management and Quantitative Methods at Università degli Studi di Milano.
    8. Andrea Hackethal & Michael Kirchler & Christine Laudenbach & Michael Razen & Annika Weber, 2020. "On the role of monetary incentives in risk preference elicitation experiments," Working Papers 2020-29, Faculty of Economics and Statistics, Universität Innsbruck.
    9. Paolo Crosetto & Antonio Filippin, 2013. "The “bomb” risk elicitation task," Journal of Risk and Uncertainty, Springer, vol. 47(1), pages 31-65, August.
    10. Paolo Crosetto & Antonio Filippin, 2013. "A Theoretical and Experimental Appraisal of Five Risk Elicitation Methods," Jena Economics Research Papers 2013-009, Friedrich-Schiller-University Jena.
    11. Michele Garagnani, 2023. "The predictive power of risk elicitation tasks," Journal of Risk and Uncertainty, Springer, vol. 67(2), pages 165-192, October.
    12. Utteeyo Dasgupta & Subha Mani & Smriti Sharma & Saurabh Singhal, 2016. "Eliciting risk preferences: Firefighting in the field," WIDER Working Paper Series wp-2016-47, World Institute for Development Economic Research (UNU-WIDER).
    13. Ihli, Hanna Julia & Chiputwa, Brian & Musshoff, Oliver, 2016. "Do Changing Probabilities or Payoffs in Lottery-Choice Experiments Affect Risk Preference Outcomes? Evidence from Rural Uganda," Journal of Agricultural and Resource Economics, Western Agricultural Economics Association, vol. 41(2), May.
    14. Jonathan P. Beauchamp & David Cesarini & Magnus Johannesson, 2017. "The psychometric and empirical properties of measures of risk preferences," Journal of Risk and Uncertainty, Springer, vol. 54(3), pages 203-237, June.
    15. Dasgupta, Utteeyo & Mani, Subha & Sharma, Smriti & Singhal, Saurabh, 2016. "Eliciting Risk Preferences: Firefighting in the Field," IZA Discussion Papers 9765, Institute of Labor Economics (IZA).
    16. Holzmeister, Felix & Stefan, Matthias, 2019. "The Risk Elicitation Puzzle Revisited: Across-Methods (In)consistency?," OSF Preprints pj9u2, Center for Open Science.
    17. Paolo Crosetto & Antonio Filippin & Janna Heider, 2015. "A Study of Outcome Reporting Bias Using Gender Differences in Risk Attitudes," CESifo Economic Studies, CESifo Group, vol. 61(1), pages 239-262.
    18. Crosetto, P. & Filippin, A., 2017. "Safe options induce gender differences in risk attitudes," Working Papers 2017-05, Grenoble Applied Economics Laboratory (GAEL).
    19. Meraner, Manuela & Musshoff, Oliver & Finger, Robert, 2018. "Using involvement to reduce inconsistencies in risk preference elicitation," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 73(C), pages 22-33.
    20. Pan He & Marcella Veronesi & Stefanie Engel, 2016. "Consistency of Risk Preference Measures and the Role of Ambiguity: An Artefactual Field Experiment from China," Working Papers 03/2016, University of Verona, Department of Economics.

    More about this item

    JEL classification:

    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • C9 - Mathematical and Quantitative Methods - - Design of Experiments
    • D8 - Microeconomics - - Information, Knowledge, and Uncertainty
    • J71 - Labor and Demographic Economics - - Labor Discrimination - - - Hiring and Firing

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:21517. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.