IDEAS home Printed from https://ideas.repec.org/p/cpr/ceprdp/13015.html
   My bibliography  Save this paper

Testing the Waters: Behavior across Participant Pools

Author

Listed:
  • Yariv, Leeat
  • Snowberg, Erik

Abstract

We leverage a large-scale incentivized survey eliciting behaviors from (almost) an entire university student population, a representative sample of the U.S. population, and Amazon Mechanical Turk (MTurk) to address concerns about the external validity of experiments with student participants. Behavior in the student population offers bounds on behaviors in other populations, and correlations between behaviors are largely similar across samples. Furthermore, non-student samples exhibit higher measurement error. Adding historical lab participation data, we find a small set of attributes over which lab participants differ from non-lab participants. Using an additional set of lab experiments, we see no evidence of observer effects.

Suggested Citation

  • Yariv, Leeat & Snowberg, Erik, 2018. "Testing the Waters: Behavior across Participant Pools," CEPR Discussion Papers 13015, C.E.P.R. Discussion Papers.
    • Handle: RePEc:cpr:ceprdp:13015
    as

    Download full text from publisher

    File URL: https://cepr.org/publications/DP13015
    Download Restriction: CEPR Discussion Papers are free to download for our researchers, subscribers and members. If you fall into one of these categories but have trouble downloading our papers, please contact us at subscribers@cepr.org
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Hoffman Elizabeth & McCabe Kevin & Shachat Keith & Smith Vernon, 1994. "Preferences, Property Rights, and Anonymity in Bargaining Games," Games and Economic Behavior, Elsevier, vol. 7(3), pages 346-380, November.
    2. Thomas Dohmen & Armin Falk & David Huffman & Uwe Sunde, 2010. "Are Risk Aversion and Impatience Related to Cognitive Ability?," American Economic Review, American Economic Association, vol. 100(3), pages 1238-1260, June.
    3. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    4. John H. Kagel & Alvin E. Roth, 2016. "The Handbook of Experimental Economics, Volume 2," Economics Books, Princeton University Press, edition 1, volume 2, number 10874.
    5. Armin Falk & James J. Heckman, 2009. "Lab Experiments are a Major Source of Knowledge in the Social Sciences," Working Papers 200935, Geary Institute, University College Dublin.
    6. Francesco Guala & Luigi Mittone, 2005. "Experiments in economics: External validity and the robustness of phenomena," Journal of Economic Methodology, Taylor & Francis Journals, vol. 12(4), pages 495-515.
    7. Muriel Niederle & Lise Vesterlund, 2007. "Do Women Shy Away From Competition? Do Men Compete Too Much?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(3), pages 1067-1101.
    8. Blair Cleave & Nikos Nikiforakis & Robert Slonim, 2013. "Is there selection bias in laboratory experiments? The case of social and risk preferences," Experimental Economics, Springer;Economic Science Association, vol. 16(3), pages 372-382, September.
    9. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    10. Belot, Michele & Duch, Raymond & Miller, Luis, 2015. "A comprehensive comparison of students and non-students in classic experimental games," Journal of Economic Behavior & Organization, Elsevier, vol. 113(C), pages 26-33.
    11. James Alm & Kim M. Bloomquist & Michael McKee, 2015. "On The External Validity Of Laboratory Tax Compliance Experiments," Economic Inquiry, Western Economic Association International, vol. 53(2), pages 1170-1186, April.
    12. Harrison, Glenn W. & Lau, Morten I. & Elisabet Rutström, E., 2009. "Risk attitudes, randomization to treatment, and self-selection into experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 70(3), pages 498-507, June.
    13. Steven D. Levitt & John A. List, 2007. "Viewpoint: On the generalizability of lab behaviour to the field," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 40(2), pages 347-370, May.
    14. Gary E. Bolton & Rami Zwick & Elena Katok, 1998. "Dictator game giving: Rules of fairness versus acts of kindness," International Journal of Game Theory, Springer;Game Theory Society, vol. 27(2), pages 269-299.
    15. Olivier Armantier & Amadou Boly, 2013. "Comparing Corruption in the Laboratory and in the Field in Burkina Faso and in Canada," Economic Journal, Royal Economic Society, vol. 123(12), pages 1168-1187, December.
    16. Steven D. Levitt & John A. List, 2007. "What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?," Journal of Economic Perspectives, American Economic Association, vol. 21(2), pages 153-174, Spring.
    17. Laury, Susan K. & Walker, James M. & Williams, Arlington W., 1995. "Anonymity and the voluntary provision of public goods," Journal of Economic Behavior & Organization, Elsevier, vol. 27(3), pages 365-380, August.
    18. Charness, Gary & Gneezy, Uri & Imas, Alex, 2013. "Experimental methods: Eliciting risk preferences," Journal of Economic Behavior & Organization, Elsevier, vol. 87(C), pages 43-51.
    19. Filippos Exadaktylos & Antonio M. Espin & Pablo Branas-Garza, 2012. "Experimental Subjects are Not Different," Working Papers 12-11, Chapman University, Economic Science Institute.
    20. Uri Gneezy, 2005. "Deception: The Role of Consequences," American Economic Review, American Economic Association, vol. 95(1), pages 384-394, March.
    21. Birk Diedenhofen & Jochen Musch, 2015. "cocor: A Comprehensive Solution for the Statistical Comparison of Correlations," PLOS ONE, Public Library of Science, vol. 10(4), pages 1-12, April.
    22. Arindrajit Dube & Jeff Jacobs & Suresh Naidu & Siddharth Suri, 2020. "Monopsony in Online Labor Markets," American Economic Review: Insights, American Economic Association, vol. 2(1), pages 33-46, March.
    23. Anna, Petrenko, 2016. "Мaркування готової продукції як складова частина інформаційного забезпечення маркетингової діяльності підприємств овочепродуктового підкомплексу," Agricultural and Resource Economics: International Scientific E-Journal, Agricultural and Resource Economics: International Scientific E-Journal, vol. 2(1), March.
    24. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    25. Arthur Schram, 2005. "Artificiality: The tension between internal and external validity in economic experiments," Journal of Economic Methodology, Taylor & Francis Journals, vol. 12(2), pages 225-237.
    26. Shane Frederick, 2005. "Cognitive Reflection and Decision Making," Journal of Economic Perspectives, American Economic Association, vol. 19(4), pages 25-42, Fall.
    27. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Engelmann, Dirk & Janeba, Eckhard & Mechtenberg, Lydia & Wehrhöfer, Nils, 2023. "Preferences over taxation of high-income individuals: Evidence from a survey experiment," European Economic Review, Elsevier, vol. 157(C).
    2. Theodor Vladasel & Simon C. Parker & Randolph Sloof & Mirjam van Praag, 2024. "Revenue drift, incentives, and effort allocation in social enterprises," Journal of Economics & Management Strategy, Wiley Blackwell, vol. 33(3), pages 630-651, August.
    3. Gary Bolton & Eugen Dimant & Ulrich Schmidt, 2018. "When a Nudge Backfires. Using Observation with Social and Economic Incentives to Promote Pro-Social Behavior," PPE Working Papers 0017, Philosophy, Politics and Economics, University of Pennsylvania.
    4. Dimant, Eugen & van Kleef, Gerben A. & Shalvi, Shaul, 2020. "Requiem for a Nudge: Framing effects in nudging honesty," Journal of Economic Behavior & Organization, Elsevier, vol. 172(C), pages 247-266.
    5. Bicchieri, Cristina & Dimant, Eugen & Xiao, Erte, 2021. "Deviant or wrong? The effects of norm information on the efficacy of punishment," Journal of Economic Behavior & Organization, Elsevier, vol. 188(C), pages 209-235.
    6. Müller, Stephan & Rau, Holger A., 2021. "Economic preferences and compliance in the social stress test of the COVID-19 crisis," Journal of Public Economics, Elsevier, vol. 194(C).
    7. Potrafke, Niklas, 2019. "Risk aversion, patience and intelligence: Evidence based on macro data," Economics Letters, Elsevier, vol. 178(C), pages 116-120.
    8. Gagnon, Nickolas & Bosmans, Kristof & Riedl, Arno, 2020. "The Effect of Unfair Chances and Gender Discrimination on Labor Supply," Research Memorandum 005, Maastricht University, Graduate School of Business and Economics (GSBE).
    9. Page, Lionel & Sarkar, Dipanwita & Silva-Goncalves, Juliana, 2019. "Long-lasting effects of relative age at school," Journal of Economic Behavior & Organization, Elsevier, vol. 168(C), pages 166-195.
    10. Jonathan Schulz & Uwe Sunde & Petra Thiemann & Christian Thoeni, 2019. "Selection into Experiments: Evidence from a Population of Students," Discussion Papers 2019-09, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    11. Salvatore Nunnari & Giovanni Montari, 2019. "Audi Alteram Partem: An Experiment on Selective Exposure to Information," Working Papers 650, IGIER (Innocenzo Gasparini Institute for Economic Research), Bocconi University.
    12. Stefano DellaVigna & Devin Pope, 2022. "Stability of Experimental Results: Forecasts and Evidence," American Economic Journal: Microeconomics, American Economic Association, vol. 14(3), pages 889-925, August.
    13. Herz, Holger & Kistler, Deborah & Zehnder, Christian & Zihlmann, Christian, 2022. "Hindsight Bias and Trust in Government: Evidence from the United States," FSES Working Papers 526, Faculty of Economics and Social Sciences, University of Freiburg/Fribourg Switzerland.
    14. Nathan Barrymore & Cristian L. Dezső & Benjamin C. King, 2022. "Gender and competitiveness when earning for others: Experimental evidence and implications for sponsorship," Strategic Management Journal, Wiley Blackwell, vol. 43(5), pages 905-934, May.
    15. John Ifcher & Homa Zarghamee & Dan Houser & Lina Diaz, 2020. "The relative income effect: an experiment," Experimental Economics, Springer;Economic Science Association, vol. 23(4), pages 1205-1234, December.
    16. Jonathan Chapman & Erik Snowberg & Stephanie Wang & Colin Camerer, 2018. "Loss Attitudes in the U.S. Population: Evidence from Dynamically Optimized Sequential Experimentation (DOSE)," CESifo Working Paper Series 7262, CESifo.
    17. Holger Herz & Deborah Kistler & Christian Zehnder & Christian Zihlmann, 2022. "Hindsight Bias and Trust in Government," CESifo Working Paper Series 9767, CESifo.
    18. Kaisa Kotakorpi & Satu Metsälampi & Topi Miettinen & Tuomas Nurminen, 2019. "The effect of reporting institutions on tax evasion:Evidence from the lab," Discussion Papers 127, Aboa Centre for Economics.
    19. Drichoutis, Andreas C. & Grimm, Veronika & Karakostas, Alexandros, 2020. "Bribing to Queue-Jump: An experiment on cultural differences in bribing attitudes among Greeks and Germans," MPRA Paper 102775, University Library of Munich, Germany.
    20. Goeschl, Timo & Kettner, Sara Elisa & Lohse, Johannes & Schwieren, Christiane, 2020. "How much can we learn about voluntary climate action from behavior in public goods games?," Ecological Economics, Elsevier, vol. 171(C).
    21. Shachat, Jason & Walker, Matthew J. & Wei, Lijia, 2021. "How the onset of the Covid-19 pandemic impacted pro-social behaviour and individual preferences: Experimental evidence from China," Journal of Economic Behavior & Organization, Elsevier, vol. 190(C), pages 480-494.
    22. Gary E. Bolton & Eugen Dimant & Ulrich Schmidt, 2020. "When a Nudge Backfires: Combining (Im)Plausible Deniability with Social and Economic Incentives to Promote Behavioral Change," CESifo Working Paper Series 8070, CESifo.
    23. Batrancea, Larissa M. & Kudła, Janusz & Błaszczak, Barbara & Kopyt, Mateusz, 2022. "Differences in tax evasion attitudes between students and entrepreneurs under the slippery slope framework," Journal of Economic Behavior & Organization, Elsevier, vol. 200(C), pages 464-482.
    24. Marcus Giamattei & Kyanoush Seyed Yahosseini & Simon Gächter & Lucas Molleman, 2020. "LIONESS Lab: a free web-based platform for conducting interactive experiments online," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(1), pages 95-111, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Erik Snowberg & Leeat Yariv, 2018. "Testing the Waters: Behavior across Participant Pools," CESifo Working Paper Series 7136, CESifo.
    2. Matteo M. Galizzi & Daniel Navarro-Martinez, 2019. "On the External Validity of Social Preference Games: A Systematic Lab-Field Study," Management Science, INFORMS, vol. 65(3), pages 976-1002, March.
    3. Jonathan Schulz & Uwe Sunde & Petra Thiemann & Christian Thoeni, 2019. "Selection into Experiments: Evidence from a Population of Students," Discussion Papers 2019-09, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    4. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    5. Jan Stoop, 2014. "From the lab to the field: envelopes, dictators and manners," Experimental Economics, Springer;Economic Science Association, vol. 17(2), pages 304-313, June.
    6. Werner, Peter & Riedl, Arno, 2018. "The role of experiments for policy design," Research Memorandum 022, Maastricht University, Graduate School of Business and Economics (GSBE).
    7. Goeschl, Timo & Kettner, Sara Elisa & Lohse, Johannes & Schwieren, Christiane, 2020. "How much can we learn about voluntary climate action from behavior in public goods games?," Ecological Economics, Elsevier, vol. 171(C).
    8. Aurélie Dariel & Nikos Nikiforakis & Jan Stoop, 2020. "Does selection bias cause us to overestimate gender differences in competitiveness?," Working Papers 20200046, New York University Abu Dhabi, Department of Social Science, revised May 2020.
    9. Proestakis, Antonios & Marandola, Ginevra & Lourenço, Joana S. & van Bavel, René, 2024. "Testing a policy intervention in the lab: differences between students and non-students in switching bank accounts," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 111(C).
    10. Barmettler, Franziska & Fehr, Ernst & Zehnder, Christian, 2012. "Big experimenter is watching you! Anonymity and prosocial behavior in the laboratory," Games and Economic Behavior, Elsevier, vol. 75(1), pages 17-34.
    11. Omar Al-Ubaydli & John A. List, 2015. "Do Natural Field Experiments Afford Researchers More or Less Control than Laboratory Experiments? A Simple Model," NBER Working Papers 20877, National Bureau of Economic Research, Inc.
    12. Johannes Abeler & Daniele Nosenzo, 2015. "Self-selection into laboratory experiments: pro-social motives versus monetary incentives," Experimental Economics, Springer;Economic Science Association, vol. 18(2), pages 195-214, June.
    13. Thiemann, Petra & Schulz, Jonathan & Sunde, Uwe & Thöni, Christian, 2022. "Selection into experiments: New evidence on the role of preferences, cognition, and recruitment protocols," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 98(C).
    14. Zhixin Dai & Fabio Galeotti & Marie Claire Villeval, 2018. "Cheating in the Lab Predicts Fraud in the Field: An Experiment in Public Transportation," Management Science, INFORMS, vol. 64(3), pages 1081-1100, March.
    15. Jean-Louis Bago & Bruce Shearer, 2022. "Risk preferences and contract choices," Experimental Economics, Springer;Economic Science Association, vol. 25(5), pages 1374-1398, November.
    16. Ben Gillen & Erik Snowberg & Leeat Yariv, 2015. "Experimenting with Measurement Error: Techniques with Applications to the Caltech Cohort Study," NBER Working Papers 21517, National Bureau of Economic Research, Inc.
    17. Murnighan, J. Keith & Wang, Long, 2016. "The social world as an experimental game," Organizational Behavior and Human Decision Processes, Elsevier, vol. 136(C), pages 80-94.
    18. Slonim, Robert & Wang, Carmen & Garbarino, Ellen & Merrett, Danielle, 2013. "Opting-in: Participation bias in economic experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 43-70.
    19. Aurelie Dariel & Nikos Nikiforakis & Jan Stoop, 2022. "Competition, Selection Bias and Gender Differences Among Economics Majors," Working Papers 20220074, New York University Abu Dhabi, Department of Social Science, revised Jan 2022.
    20. Abel Brodeur, Nikolai M. Cook, Anthony Heyes, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell Us about Publication Bias and p-Hacking in Online Experiments," LCERPA Working Papers am0133, Laurier Centre for Economic Research and Policy Analysis.

    More about this item

    Keywords

    Lab selection; External validity; Experiments;
    All these keywords.

    JEL classification:

    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C80 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - General
    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cpr:ceprdp:13015. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://www.cepr.org .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.