IDEAS home Printed from https://ideas.repec.org/p/ces/ceswps/_7136.html
   My bibliography  Save this paper

Testing the Waters: Behavior across Participant Pools

Author

Listed:
  • Erik Snowberg
  • Leeat Yariv

Abstract

We leverage a large-scale incentivized survey eliciting behaviors from (almost) an entire university student population, a representative sample of the U.S. population, and Amazon Mechanical Turk (MTurk) to address concerns about the external validity of experiments with student participants. Behavior in the student population offers bounds on behaviors in other populations, and correlations between behaviors are largely similar across samples. Furthermore, non-student samples exhibit higher measurement error. Adding historical lab participation data, we find a small set of attributes over which lab participants differ from non-lab participants. Using an additional set of lab experiments, we see no evidence of observer effects.

Suggested Citation

  • Erik Snowberg & Leeat Yariv, 2018. "Testing the Waters: Behavior across Participant Pools," CESifo Working Paper Series 7136, CESifo.
    • Handle: RePEc:ces:ceswps:_7136
    as

    Download full text from publisher

    File URL: https://www.cesifo.org/DocDL/cesifo1_wp7136.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Hoffman Elizabeth & McCabe Kevin & Shachat Keith & Smith Vernon, 1994. "Preferences, Property Rights, and Anonymity in Bargaining Games," Games and Economic Behavior, Elsevier, vol. 7(3), pages 346-380, November.
    2. Thomas Dohmen & Armin Falk & David Huffman & Uwe Sunde, 2010. "Are Risk Aversion and Impatience Related to Cognitive Ability?," American Economic Review, American Economic Association, vol. 100(3), pages 1238-1260, June.
    3. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    4. Armin Falk & James J. Heckman, 2009. "Lab Experiments are a Major Source of Knowledge in the Social Sciences," Working Papers 200935, Geary Institute, University College Dublin.
    5. Frechette, Guillaume R. & Schotter, Andrew (ed.), 2015. "Handbook of Experimental Economic Methodology," OUP Catalogue, Oxford University Press, number 9780195328325.
    6. Arindrajit Dube & Jeff Jacobs & Suresh Naidu & Siddharth Suri, 2020. "Monopsony in Online Labor Markets," American Economic Review: Insights, American Economic Association, vol. 2(1), pages 33-46, March.
    7. Belot, Michele & Duch, Raymond & Miller, Luis, 2015. "A comprehensive comparison of students and non-students in classic experimental games," Journal of Economic Behavior & Organization, Elsevier, vol. 113(C), pages 26-33.
    8. Arthur Schram, 2005. "Artificiality: The tension between internal and external validity in economic experiments," Journal of Economic Methodology, Taylor & Francis Journals, vol. 12(2), pages 225-237.
    9. Shane Frederick, 2005. "Cognitive Reflection and Decision Making," Journal of Economic Perspectives, American Economic Association, vol. 19(4), pages 25-42, Fall.
    10. James Alm & Kim M. Bloomquist & Michael McKee, 2015. "On The External Validity Of Laboratory Tax Compliance Experiments," Economic Inquiry, Western Economic Association International, vol. 53(2), pages 1170-1186, April.
    11. Uri Gneezy, 2005. "Deception: The Role of Consequences," American Economic Review, American Economic Association, vol. 95(1), pages 384-394, March.
    12. Harrison, Glenn W. & Lau, Morten I. & Elisabet Rutström, E., 2009. "Risk attitudes, randomization to treatment, and self-selection into experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 70(3), pages 498-507, June.
    13. Steven D. Levitt & John A. List, 2007. "Viewpoint: On the generalizability of lab behaviour to the field," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 40(2), pages 347-370, May.
    14. Francesco Guala & Luigi Mittone, 2005. "Experiments in economics: External validity and the robustness of phenomena," Journal of Economic Methodology, Taylor & Francis Journals, vol. 12(4), pages 495-515.
    15. Gary E. Bolton & Rami Zwick & Elena Katok, 1998. "Dictator game giving: Rules of fairness versus acts of kindness," International Journal of Game Theory, Springer;Game Theory Society, vol. 27(2), pages 269-299.
    16. Blair Cleave & Nikos Nikiforakis & Robert Slonim, 2013. "Is there selection bias in laboratory experiments? The case of social and risk preferences," Experimental Economics, Springer;Economic Science Association, vol. 16(3), pages 372-382, September.
    17. Birk Diedenhofen & Jochen Musch, 2015. "cocor: A Comprehensive Solution for the Statistical Comparison of Correlations," PLOS ONE, Public Library of Science, vol. 10(4), pages 1-12, April.
    18. Muriel Niederle & Lise Vesterlund, 2007. "Do Women Shy Away From Competition? Do Men Compete Too Much?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(3), pages 1067-1101.
    19. Anna, Petrenko, 2016. "Мaркування готової продукції як складова частина інформаційного забезпечення маркетингової діяльності підприємств овочепродуктового підкомплексу," Agricultural and Resource Economics: International Scientific E-Journal, Agricultural and Resource Economics: International Scientific E-Journal, vol. 2(1), March.
    20. Olivier Armantier & Amadou Boly, 2013. "Comparing Corruption in the Laboratory and in the Field in Burkina Faso and in Canada," Economic Journal, Royal Economic Society, vol. 123(12), pages 1168-1187, December.
    21. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    22. Steven D. Levitt & John A. List, 2007. "What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?," Journal of Economic Perspectives, American Economic Association, vol. 21(2), pages 153-174, Spring.
    23. Laury, Susan K. & Walker, James M. & Williams, Arlington W., 1995. "Anonymity and the voluntary provision of public goods," Journal of Economic Behavior & Organization, Elsevier, vol. 27(3), pages 365-380, August.
    24. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    25. Charness, Gary & Gneezy, Uri & Imas, Alex, 2013. "Experimental methods: Eliciting risk preferences," Journal of Economic Behavior & Organization, Elsevier, vol. 87(C), pages 43-51.
    26. Filippos Exadaktylos & Antonio M. Espin & Pablo Branas-Garza, 2012. "Experimental Subjects are Not Different," Working Papers 12-11, Chapman University, Economic Science Institute.
    27. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Engelmann, Dirk & Janeba, Eckhard & Mechtenberg, Lydia & Wehrhöfer, Nils, 2023. "Preferences over taxation of high-income individuals: Evidence from a survey experiment," European Economic Review, Elsevier, vol. 157(C).
    2. Nathan Barrymore & Cristian L. Dezső & Benjamin C. King, 2022. "Gender and competitiveness when earning for others: Experimental evidence and implications for sponsorship," Strategic Management Journal, Wiley Blackwell, vol. 43(5), pages 905-934, May.
    3. Goeschl, Timo & Kettner, Sara Elisa & Lohse, Johannes & Schwieren, Christiane, 2020. "How much can we learn about voluntary climate action from behavior in public goods games?," Ecological Economics, Elsevier, vol. 171(C).
    4. Jonathan Schulz & Uwe Sunde & Petra Thiemann & Christian Thoeni, 2019. "Selection into Experiments: Evidence from a Population of Students," Discussion Papers 2019-09, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    5. Gary Bolton & Eugen Dimant & Ulrich Schmidt, 2018. "When a Nudge Backfires. Using Observation with Social and Economic Incentives to Promote Pro-Social Behavior," PPE Working Papers 0017, Philosophy, Politics and Economics, University of Pennsylvania.
    6. Shachat, Jason & Walker, Matthew J. & Wei, Lijia, 2021. "How the onset of the Covid-19 pandemic impacted pro-social behaviour and individual preferences: Experimental evidence from China," Journal of Economic Behavior & Organization, Elsevier, vol. 190(C), pages 480-494.
    7. John Ifcher & Homa Zarghamee & Dan Houser & Lina Diaz, 2020. "The relative income effect: an experiment," Experimental Economics, Springer;Economic Science Association, vol. 23(4), pages 1205-1234, December.
    8. Dimant, Eugen & van Kleef, Gerben A. & Shalvi, Shaul, 2020. "Requiem for a Nudge: Framing effects in nudging honesty," Journal of Economic Behavior & Organization, Elsevier, vol. 172(C), pages 247-266.
    9. Stefano DellaVigna & Devin Pope, 2022. "Stability of Experimental Results: Forecasts and Evidence," American Economic Journal: Microeconomics, American Economic Association, vol. 14(3), pages 889-925, August.
    10. Vladasel, Theodor & Parker, Simon C. & Sloof, Randolph & van Praag, Mirjam C., 2022. "Revenue Drift, Incentives, and Effort Allocation in Social Enterprises," IZA Discussion Papers 15716, Institute of Labor Economics (IZA).
    11. Bicchieri, Cristina & Dimant, Eugen & Xiao, Erte, 2021. "Deviant or wrong? The effects of norm information on the efficacy of punishment," Journal of Economic Behavior & Organization, Elsevier, vol. 188(C), pages 209-235.
    12. Salvatore Nunnari & Giovanni Montari, 2019. "Audi Alteram Partem: An Experiment on Selective Exposure to Information," Working Papers 650, IGIER (Innocenzo Gasparini Institute for Economic Research), Bocconi University.
    13. Jonathan Chapman & Erik Snowberg & Stephanie Wang & Colin Camerer, 2018. "Loss Attitudes in the U.S. Population: Evidence from Dynamically Optimized Sequential Experimentation (DOSE)," CESifo Working Paper Series 7262, CESifo.
    14. Potrafke, Niklas, 2019. "Risk aversion, patience and intelligence: Evidence based on macro data," Economics Letters, Elsevier, vol. 178(C), pages 116-120.
    15. Gagnon, Nickolas & Bosmans, Kristof & Riedl, Arno, 2020. "The Effect of Unfair Chances and Gender Discrimination on Labor Supply," IZA Discussion Papers 12912, Institute of Labor Economics (IZA).
    16. Müller, Stephan & Rau, Holger A., 2021. "Economic preferences and compliance in the social stress test of the COVID-19 crisis," Journal of Public Economics, Elsevier, vol. 194(C).
    17. Andreas C. Drichoutis & Veronika Grimm & Alexandros Karakostas, 2020. "Bribing to Queue-Jump: An experiment on cultural differences in bribing attitudes among Greeks and Germans," Working Papers 2020-2, Agricultural University of Athens, Department Of Agricultural Economics.
    18. Gary E. Bolton & Eugen Dimant & Ulrich Schmidt, 2020. "When a Nudge Backfires: Combining (Im)Plausible Deniability with Social and Economic Incentives to Promote Behavioral Change," CESifo Working Paper Series 8070, CESifo.
    19. Page, Lionel & Sarkar, Dipanwita & Silva-Goncalves, Juliana, 2019. "Long-lasting effects of relative age at school," Journal of Economic Behavior & Organization, Elsevier, vol. 168(C), pages 166-195.
    20. Batrancea, Larissa M. & Kudła, Janusz & Błaszczak, Barbara & Kopyt, Mateusz, 2022. "Differences in tax evasion attitudes between students and entrepreneurs under the slippery slope framework," Journal of Economic Behavior & Organization, Elsevier, vol. 200(C), pages 464-482.
    21. Holger Herz & Deborah Kistler & Christian Zehnder & Christian Zihlmann, 2022. "Hindsight Bias and Trust in Government," CESifo Working Paper Series 9767, CESifo.
    22. Herz, Holger & Kistler, Deborah & Zehnder, Christian & Zihlmann, Christian, 2022. "Hindsight Bias and Trust in Government: Evidence from the United States," FSES Working Papers 526, Faculty of Economics and Social Sciences, University of Freiburg/Fribourg Switzerland.
    23. Marcus Giamattei & Kyanoush Seyed Yahosseini & Simon Gächter & Lucas Molleman, 2020. "LIONESS Lab: a free web-based platform for conducting interactive experiments online," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(1), pages 95-111, June.
    24. Kaisa Kotakorpi & Satu Metsälampi & Topi Miettinen & Tuomas Nurminen, 2019. "The effect of reporting institutions on tax evasion:Evidence from the lab," Discussion Papers 127, Aboa Centre for Economics.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yariv, Leeat & Snowberg, Erik, 2018. "Testing the Waters: Behavior across Participant Pools," CEPR Discussion Papers 13015, C.E.P.R. Discussion Papers.
    2. Matteo M. Galizzi & Daniel Navarro-Martinez, 2019. "On the External Validity of Social Preference Games: A Systematic Lab-Field Study," Management Science, INFORMS, vol. 65(3), pages 976-1002, March.
    3. Jonathan Schulz & Uwe Sunde & Petra Thiemann & Christian Thoeni, 2019. "Selection into Experiments: Evidence from a Population of Students," Discussion Papers 2019-09, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    4. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    5. Jan Stoop, 2014. "From the lab to the field: envelopes, dictators and manners," Experimental Economics, Springer;Economic Science Association, vol. 17(2), pages 304-313, June.
    6. Zhixin Dai & Fabio Galeotti & Marie Claire Villeval, 2018. "Cheating in the Lab Predicts Fraud in the Field: An Experiment in Public Transportation," Management Science, INFORMS, vol. 64(3), pages 1081-1100, March.
    7. Goeschl, Timo & Kettner, Sara Elisa & Lohse, Johannes & Schwieren, Christiane, 2020. "How much can we learn about voluntary climate action from behavior in public goods games?," Ecological Economics, Elsevier, vol. 171(C).
    8. Aurélie Dariel & Nikos Nikiforakis & Jan Stoop, 2020. "Does selection bias cause us to overestimate gender differences in competitiveness?," Working Papers 20200046, New York University Abu Dhabi, Department of Social Science, revised May 2020.
    9. Barmettler, Franziska & Fehr, Ernst & Zehnder, Christian, 2012. "Big experimenter is watching you! Anonymity and prosocial behavior in the laboratory," Games and Economic Behavior, Elsevier, vol. 75(1), pages 17-34.
    10. Omar Al-Ubaydli & John A. List, 2015. "Do Natural Field Experiments Afford Researchers More or Less Control than Laboratory Experiments? A Simple Model," NBER Working Papers 20877, National Bureau of Economic Research, Inc.
    11. Johannes Abeler & Daniele Nosenzo, 2015. "Self-selection into laboratory experiments: pro-social motives versus monetary incentives," Experimental Economics, Springer;Economic Science Association, vol. 18(2), pages 195-214, June.
    12. Thiemann, Petra & Schulz, Jonathan & Sunde, Uwe & Thöni, Christian, 2022. "Selection into experiments: New evidence on the role of preferences, cognition, and recruitment protocols," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 98(C).
    13. Jean-Louis Bago & Bruce Shearer, 2022. "Risk preferences and contract choices," Experimental Economics, Springer;Economic Science Association, vol. 25(5), pages 1374-1398, November.
    14. Ben Gillen & Erik Snowberg & Leeat Yariv, 2015. "Experimenting with Measurement Error: Techniques with Applications to the Caltech Cohort Study," NBER Working Papers 21517, National Bureau of Economic Research, Inc.
    15. Murnighan, J. Keith & Wang, Long, 2016. "The social world as an experimental game," Organizational Behavior and Human Decision Processes, Elsevier, vol. 136(C), pages 80-94.
    16. Slonim, Robert & Wang, Carmen & Garbarino, Ellen & Merrett, Danielle, 2013. "Opting-in: Participation bias in economic experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 43-70.
    17. Werner, Peter & Riedl, Arno, 2018. "The role of experiments for policy design," Research Memorandum 022, Maastricht University, Graduate School of Business and Economics (GSBE).
    18. Aurelie Dariel & Nikos Nikiforakis & Jan Stoop, 2022. "Competition, Selection Bias and Gender Differences Among Economics Majors," Working Papers 20220074, New York University Abu Dhabi, Department of Social Science, revised Jan 2022.
    19. Sébastien Duchêne & Adrien Nguyen-Huu & Dimitri Dubois & Marc Willinger, 2022. "Risk-return trade-offs in the context of environmental impact: a lab-in-the-field experiment with finance professionals," CEE-M Working Papers hal-03883121, CEE-M, Universtiy of Montpellier, CNRS, INRA, Montpellier SupAgro.
    20. Abel Brodeur, Nikolai M. Cook, Anthony Heyes, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell Us about Publication Bias and p-Hacking in Online Experiments," LCERPA Working Papers am0133, Laurier Centre for Economic Research and Policy Analysis.

    More about this item

    Keywords

    lab selection; external validity; experiments;
    All these keywords.

    JEL classification:

    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C80 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - General
    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ces:ceswps:_7136. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Klaus Wohlrabe (email available below). General contact details of provider: https://edirc.repec.org/data/cesifde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.