IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/21517.html
   My bibliography  Save this paper

Experimenting with Measurement Error: Techniques with Applications to the Caltech Cohort Study

Author

Listed:
  • Ben Gillen
  • Erik Snowberg
  • Leeat Yariv

Abstract

Measurement error is ubiquitous in experimental work. It leads to imperfect statistical controls, attenuated estimated effects of elicited behaviors, and biased correlations between characteristics. We develop simple statistical techniques for dealing with experimental measurement error. These techniques are applied to data from the Caltech Cohort Study, which conducts repeated incentivized surveys of the Caltech student body. We illustrate the impact of measurement error by replicating three classic experiments, and showing that results change substantially when measurement error is taken into account. Collectively, these results show that failing to properly account for measurement error may cause a field-wide bias: it may lead scholars to identify "new" effects and phenomena that are actually similar to those previously documented.

Suggested Citation

  • Ben Gillen & Erik Snowberg & Leeat Yariv, 2015. "Experimenting with Measurement Error: Techniques with Applications to the Caltech Cohort Study," NBER Working Papers 21517, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:21517
    Note: DEV LS POL
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w21517.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Alan B. Krueger, 2001. "Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 69-85, Fall.
    2. Catherine C. Eckel & Philip J. Grossman, 2002. "Sex Differences and Statistical Stereotyping in Attitudes Toward Financial Risk," Monash Economics Working Papers archive-03, Monash University, Department of Economics.
    3. Uri Gneezy & Jan Potters, 1997. "An Experiment on Risk Taking and Evaluation Periods," The Quarterly Journal of Economics, Oxford University Press, vol. 112(2), pages 631-645.
    4. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    5. Jerry Hausman, 2001. "Mismeasured Variables in Econometric Analysis: Problems from the Right and Problems from the Left," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 57-67, Fall.
    6. Blattman, Christopher & Jamison, Julian & Koroknay-Palicz, Tricia & Rodrigues, Katherine & Sheridan, Margaret, 2016. "Measuring the measurement error: A method to qualitatively validate survey data," Journal of Development Economics, Elsevier, vol. 120(C), pages 99-112.
    7. Lisa Anderson & Jennifer Mellor, 2009. "Are risk preferences stable? Comparing an experimental measure with a validated survey-based measure," Journal of Risk and Uncertainty, Springer, vol. 39(2), pages 137-160, October.
    8. Alexandre Belloni & Victor Chernozhukov & Christian Hansen, 2011. "Inference on Treatment Effects After Selection Amongst High-Dimensional Controls," Papers 1201.0224, arXiv.org, revised May 2012.
    9. Segal, Uzi, 1987. "The Ellsberg Paradox and Risk Aversion: An Anticipated Utility Approach," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 28(1), pages 175-202, February.
    10. Milton Friedman, 1957. "Introduction to "A Theory of the Consumption Function"," NBER Chapters, in: A Theory of the Consumption Function, pages 1-6, National Bureau of Economic Research, Inc.
    11. Bound, John & Brown, Charles & Mathiowetz, Nancy, 2001. "Measurement error in survey data," Handbook of Econometrics, in: J.J. Heckman & E.E. Leamer (ed.), Handbook of Econometrics, edition 1, volume 5, chapter 59, pages 3705-3843, Elsevier.
    12. Hertz Tom & Jayasundera Tamara & Piraino Patrizio & Selcuk Sibel & Smith Nicole & Verashchagina Alina, 2008. "The Inheritance of Educational Inequality: International Comparisons and Fifty-Year Trends," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 7(2), pages 1-48, January.
    13. Thomas Dohmen & Armin Falk & David Huffman & Uwe Sunde & Jürgen Schupp & Gert G. Wagner, 2011. "Individual Risk Attitudes: Measurement, Determinants, And Behavioral Consequences," Journal of the European Economic Association, European Economic Association, vol. 9(3), pages 522-550, June.
    14. Leeb, Hannes & Pötscher, Benedikt M., 2005. "Model Selection And Inference: Facts And Fiction," Econometric Theory, Cambridge University Press, vol. 21(1), pages 21-59, February.
    15. McKelvey Richard D. & Palfrey Thomas R., 1995. "Quantal Response Equilibria for Normal Form Games," Games and Economic Behavior, Elsevier, vol. 10(1), pages 6-38, July.
    16. Fan, Jianqing & Liao, Yuan, 2012. "Endogeneity in ultrahigh dimension," MPRA Paper 38698, University Library of Munich, Germany.
    17. Liran Einav & Amy Finkelstein & Iuliana Pascu & Mark R. Cullen, 2012. "How General Are Risk Preferences? Choices under Uncertainty in Different Domains," American Economic Review, American Economic Association, vol. 102(6), pages 2606-2638, October.
    18. Charles A. Holt & Susan K. Laury, 2002. "Risk Aversion and Incentive Effects," American Economic Review, American Economic Association, vol. 92(5), pages 1644-1655, December.
    19. Harless, David W & Camerer, Colin F, 1994. "The Predictive Utility of Generalized Expected Utility Theories," Econometrica, Econometric Society, vol. 62(6), pages 1251-1289, November.
    20. Eckel, Catherine C. & Grossman, Philip J., 2008. "Men, Women and Risk Aversion: Experimental Evidence," Handbook of Experimental Economics Results, in: Charles R. Plott & Vernon L. Smith (ed.), Handbook of Experimental Economics Results, edition 1, volume 1, chapter 113, pages 1061-1073, Elsevier.
    21. Christoph Engel, 2011. "Dictator games: a meta study," Experimental Economics, Springer;Economic Science Association, vol. 14(4), pages 583-610, November.
    22. Blair Cleave & Nikos Nikiforakis & Robert Slonim, 2013. "Is there selection bias in laboratory experiments? The case of social and risk preferences," Experimental Economics, Springer;Economic Science Association, vol. 16(3), pages 372-382, September.
    23. Nagel, Rosemarie, 1995. "Unraveling in Guessing Games: An Experimental Study," American Economic Review, American Economic Association, vol. 85(5), pages 1313-1326, December.
    24. Fan J. & Li R., 2001. "Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties," Journal of the American Statistical Association, American Statistical Association, vol. 96, pages 1348-1360, December.
    25. repec:fth:prinin:455 is not listed on IDEAS
    26. G.M. Constantinides & M. Harris & R. M. Stulz (ed.), 2013. "Handbook of the Economics of Finance," Handbook of the Economics of Finance, Elsevier, volume 2, number 2-b.
    27. G.M. Constantinides & M. Harris & R. M. Stulz (ed.), 2013. "Handbook of the Economics of Finance," Handbook of the Economics of Finance, Elsevier, volume 2, number 2-a.
    28. A. Belloni & V. Chernozhukov & L. Wang, 2011. "Square-root lasso: pivotal recovery of sparse signals via conic programming," Biometrika, Biometrika Trust, vol. 98(4), pages 791-806.
    29. Barber, Brad M. & Odean, Terrance, 2013. "The Behavior of Individual Investors," Handbook of the Economics of Finance, in: G.M. Constantinides & M. Harris & R. M. Stulz (ed.), Handbook of the Economics of Finance, volume 2, chapter 0, pages 1533-1570, Elsevier.
    30. Shane Frederick, 2005. "Cognitive Reflection and Decision Making," Journal of Economic Perspectives, American Economic Association, vol. 19(4), pages 25-42, Fall.
    31. Harrison, Glenn W. & Lau, Morten I. & Elisabet Rutström, E., 2009. "Risk attitudes, randomization to treatment, and self-selection into experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 70(3), pages 498-507, June.
    32. Yoram Halevy, 2007. "Ellsberg Revisited: An Experimental Study," Econometrica, Econometric Society, vol. 75(2), pages 503-536, March.
    33. Milton Friedman, 1957. "A Theory of the Consumption Function," NBER Books, National Bureau of Economic Research, Inc, number frie57-1.
    34. Joshua Angrist & Alan Krueger, 2001. "Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments," Working Papers 834, Princeton University, Department of Economics, Industrial Relations Section..
    35. Andrews, Donald W K, 2001. "Testing When a Parameter Is on the Boundary of the Maintained Hypothesis," Econometrica, Econometric Society, vol. 69(3), pages 683-734, May.
    36. Sendhil Mullainathan & Marianne Bertrand, 2001. "Do People Mean What They Say? Implications for Subjective Survey Data," American Economic Review, American Economic Association, vol. 91(2), pages 67-72, May.
    37. Armin Falk & Stephan Meier & Christian Zehnder, 2013. "Do Lab Experiments Misrepresent Social Preferences? The Case Of Self-Selected Student Samples," Journal of the European Economic Association, European Economic Association, vol. 11(4), pages 839-852, August.
    38. Friedman, Daniel & Isaac, R. Mark & James, Duncan & Sunder, Shyam, 2014. "Risky Curves: On the Empirical Failure of Expected Utility," Santa Cruz Department of Economics, Working Paper Series qt87v8k86z, Department of Economics, UC Santa Cruz.
    39. Charness, Gary & Gneezy, Uri & Imas, Alex, 2013. "Experimental methods: Eliciting risk preferences," Journal of Economic Behavior & Organization, Elsevier, vol. 87(C), pages 43-51.
    40. Rachel Croson & Uri Gneezy, 2009. "Gender Differences in Preferences," Journal of Economic Literature, American Economic Association, vol. 47(2), pages 448-474, June.
    41. Chetan Dave & Catherine Eckel & Cathleen Johnson & Christian Rojas, 2010. "Eliciting risk preferences: When is simple better?," Journal of Risk and Uncertainty, Springer, vol. 41(3), pages 219-243, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Stefano Giglio & Matteo Maggiori & Johannes Stroebel & Stephen Utkus, 2021. "Five Facts about Beliefs and Portfolios," American Economic Review, American Economic Association, vol. 111(5), pages 1481-1522, May.
    2. Buser, Thomas & Ranehill, Eva & van Veldhuizen, Roel, 2021. "Gender differences in willingness to compete: The role of public observability," Journal of Economic Psychology, Elsevier, vol. 83(C).
    3. Dietmar Fehr & Julia Schmid, 2018. "Exclusion in all‐pay auctions: An experimental investigation," Journal of Economics & Management Strategy, Wiley Blackwell, vol. 27(2), pages 326-339, June.
    4. Thomas Buser & Huaiping Yuan, 2019. "Do Women Give Up Competing More Easily? Evidence from the Lab and the Dutch Math Olympiad," American Economic Journal: Applied Economics, American Economic Association, vol. 11(3), pages 225-252, July.
    5. Flory, Jeffrey A. & Gneezy, Uri & Leonard, Kenneth L. & List, John A., 2018. "Gender, age, and competition: A disappearing gap?," Journal of Economic Behavior & Organization, Elsevier, vol. 150(C), pages 256-276.
    6. Filippin, Antonio & Gioia, Francesca, 2018. "Competition and subsequent risk-taking behaviour: Heterogeneity across gender and outcomes," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 75(C), pages 84-94.
    7. Roman Inderst & Kiryl Khalmetski & Axel Ockenfels, 2019. "Sharing Guilt: How Better Access to Information May Backfire," Management Science, INFORMS, vol. 65(7), pages 3322-3336, July.
    8. Stone, Daniel, 2018. ""Unmotivated Bias" and Partisan Hostility: Empirical Evidence," SocArXiv hr5ba, Center for Open Science.
    9. Jonathan P. Beauchamp & David Cesarini & Magnus Johannesson, 2017. "The psychometric and empirical properties of measures of risk preferences," Journal of Risk and Uncertainty, Springer, vol. 54(3), pages 203-237, June.
    10. Carpenter, Jeffrey & Frank, Rachel & Huet-Vaughn, Emiliano, 2018. "Gender differences in interpersonal and intrapersonal competitive behavior," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 77(C), pages 170-176.
    11. Castillo, Marco & Jordan, Jeffrey L. & Petrie, Ragan, 2018. "Children’s rationality, risk attitudes and field behavior," European Economic Review, Elsevier, vol. 102(C), pages 62-81.
    12. Emel Filiz-Ozbay & John C. Ham & John H. Kagel & Erkut Y. Ozbay, 2018. "The role of cognitive ability and personality traits for men and women in gift exchange outcomes," Experimental Economics, Springer;Economic Science Association, vol. 21(3), pages 650-672, September.
    13. Benjamin Enke & Ricardo Rodríguez-Padilla & Florian Zimmermann, 2019. "Moral Universalism and the Structure of Ideology," CESifo Working Paper Series 7924, CESifo.
    14. Hunt Allcott & Benjamin B Lockwood & Dmitry Taubinsky, 2019. "Regressive Sin Taxes, with an Application to the Optimal Soda Tax," The Quarterly Journal of Economics, Oxford University Press, vol. 134(3), pages 1557-1626.
    15. Thomas Buser & Anna Dreber & Johanna Mollerstrom, 2017. "The impact of stress on tournament entry," Experimental Economics, Springer;Economic Science Association, vol. 20(2), pages 506-530, June.
    16. Thomas Buser & Louis Putterman & Joël van der Weele, 2016. "Gender and Redistribution: Experimental Evidence," Tinbergen Institute Discussion Papers 16-063/I, Tinbergen Institute.
    17. Bruno Wichmann & Peter Boxall & Scott Wilson & Orsolya Pergery, 2017. "Auctioning Risky Conservation Contracts," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 68(4), pages 1111-1144, December.
    18. Thomas Buser, 2016. "How does the Gender Difference in Willingness to Compete evolve with Experience?," Tinbergen Institute Discussion Papers 16-017/I, Tinbergen Institute.
    19. Anya Samek, 2019. "Gender Differences in Job Entry Decisions: A University-Wide Field Experiment," Management Science, INFORMS, vol. 65(7), pages 3272-3281, July.
    20. Buser, Thomas & Cappelen, Alexander & Tungodden, Bertil, 2021. "Fairness and Willingness to Compete," Discussion Paper Series in Economics 8/2021, Norwegian School of Economics, Department of Economics.
    21. Brocas, Isabelle & Carrillo, Juan D. & Tarrasó, Jorge, 2018. "How long is a minute?," Games and Economic Behavior, Elsevier, vol. 111(C), pages 305-322.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Paolo Crosetto & Antonio Filippin, 2016. "A theoretical and experimental appraisal of four risk elicitation methods," Experimental Economics, Springer;Economic Science Association, vol. 19(3), pages 613-641, September.
    2. Ranganathan, Kavitha & Lejarraga, Tomás, 2021. "Elicitation of risk preferences through satisficing," Journal of Behavioral and Experimental Finance, Elsevier, vol. 32(C).
    3. Antonio Filippin & Paolo Crosetto, 2016. "A Reconsideration of Gender Differences in Risk Attitudes," Management Science, INFORMS, vol. 62(11), pages 3138-3160, November.
    4. Galizzi, Matteo M. & Machado, Sara R. & Miniaci, Raffaele, 2016. "Temporal stability, cross-validity, and external validity of risk preferences measures: experimental evidence from a UK representative sample," LSE Research Online Documents on Economics 67554, London School of Economics and Political Science, LSE Library.
    5. Paolo Crosetto & Antonio Filippin, 2013. "The “bomb” risk elicitation task," Journal of Risk and Uncertainty, Springer, vol. 47(1), pages 31-65, August.
    6. Tamás Csermely & Alexander Rabas, 2016. "How to reveal people’s preferences: Comparing time consistency and predictive power of multiple price list risk elicitation methods," Journal of Risk and Uncertainty, Springer, vol. 53(2), pages 107-136, December.
    7. Utteeyo Dasgupta & Subha Mani & Smriti Sharma & Saurabh Singhal, 2016. "Eliciting risk preferences: Firefighting in the field," WIDER Working Paper Series wp-2016-47, World Institute for Development Economic Research (UNU-WIDER).
    8. Paolo Crosetto & Antonio Filippin, 2013. "A Theoretical and Experimental Appraisal of Five Risk Elicitation Methods," SOEPpapers on Multidisciplinary Panel Data Research 547, DIW Berlin, The German Socio-Economic Panel (SOEP).
    9. Menkhoff, Lukas & Sakha, Sahra, 2017. "Estimating risky behavior with multiple-item risk measures," Journal of Economic Psychology, Elsevier, vol. 59(C), pages 59-86.
    10. Ihli, Hanna Julia & Chiputwa, Brian & Musshoff, Oliver, 2016. "Do Changing Probabilities or Payoffs in Lottery-Choice Experiments Affect Risk Preference Outcomes? Evidence from Rural Uganda," Journal of Agricultural and Resource Economics, Western Agricultural Economics Association, vol. 41(2), May.
    11. Andrea Hackethal & Michael Kirchler & Christine Laudenbach & Michael Razen & Annika Weber, 2020. "On the role of monetary incentives in risk preference elicitation experiments," Working Papers 2020-29, Faculty of Economics and Statistics, University of Innsbruck.
    12. Jonathan P. Beauchamp & David Cesarini & Magnus Johannesson, 2017. "The psychometric and empirical properties of measures of risk preferences," Journal of Risk and Uncertainty, Springer, vol. 54(3), pages 203-237, June.
    13. Antonio FILIPPIN & Paolo CROSETTO, 2014. "A Reconsideration of Gender Differences in Risk Attitudes," Departmental Working Papers 2014-01, Department of Economics, Management and Quantitative Methods at Università degli Studi di Milano.
    14. Paolo Crosetto & Antonio Filippin & Janna Heider, 2015. "A Study of Outcome Reporting Bias Using Gender Differences in Risk Attitudes," CESifo Economic Studies, CESifo, vol. 61(1), pages 239-262.
    15. Muriel Niederle, 2014. "Gender," NBER Working Papers 20788, National Bureau of Economic Research, Inc.
    16. Dasgupta, Utteeyo & Mani, Subha & Sharma, Smriti & Singhal, Saurabh, 2016. "Eliciting Risk Preferences: Firefighting in the Field," IZA Discussion Papers 9765, Institute of Labor Economics (IZA).
    17. Holzmeister, Felix & Stefan, Matthias, 2019. "The Risk Elicitation Puzzle Revisited: Across-Methods (In)consistency?," OSF Preprints pj9u2, Center for Open Science.
    18. Felix Holzmeister & Matthias Stefan, 2019. "The risk elicitation puzzle revisited: Across-methods (in)consistency?," Working Papers 2019-19, Faculty of Economics and Statistics, University of Innsbruck.
    19. Crosetto, P. & Filippin, A., 2017. "Safe options induce gender differences in risk attitudes," Working Papers 2017-05, Grenoble Applied Economics Laboratory (GAEL).
    20. Alan, Sule & Baydar, Nazli & Boneva, Teodora & Crossley, Thomas F. & Ertac, Seda, 2017. "Transmission of risk preferences from mothers to daughters," Journal of Economic Behavior & Organization, Elsevier, vol. 134(C), pages 60-77.

    More about this item

    JEL classification:

    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • C9 - Mathematical and Quantitative Methods - - Design of Experiments
    • D8 - Microeconomics - - Information, Knowledge, and Uncertainty
    • J71 - Labor and Demographic Economics - - Labor Discrimination - - - Hiring and Firing

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:21517. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.