IDEAS home Printed from https://ideas.repec.org/a/eee/jeborg/v90y2013icp43-70.html
   My bibliography  Save this article

Opting-in: Participation bias in economic experiments

Author

Listed:
  • Slonim, Robert
  • Wang, Carmen
  • Garbarino, Ellen
  • Merrett, Danielle

Abstract

Assuming individuals rationally decide whether to participate or not to participate in lab experiments, we hypothesize several non-representative biases in the characteristics of lab participants. We test the hypotheses by first collecting survey and experimental data from a typical recruitment population and then inviting them to participate in a lab experiment. The results indicate that lab participants are not representative of the target population on almost all the hypothesized characteristics, including having lower income, working fewer hours, volunteering more often, and exhibiting behaviors correlated with interest in experiments and economics. The results reinforce the commonly understood limits of laboratory research to make quantitative inferences. We also discuss several methods for addressing non-representative biases to advance laboratory methods for improving quantitative inferences and consequently increasing confidence in qualitative conclusions.

Suggested Citation

  • Slonim, Robert & Wang, Carmen & Garbarino, Ellen & Merrett, Danielle, 2013. "Opting-in: Participation bias in economic experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 43-70.
  • Handle: RePEc:eee:jeborg:v:90:y:2013:i:c:p:43-70 DOI: 10.1016/j.jebo.2013.03.013
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167268113000589
    Download Restriction: Full text for ScienceDirect subscribers only

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    2. Eckel, Catherine C. & Grossman, Philip J., 1996. "Altruism in Anonymous Dictator Games," Games and Economic Behavior, Elsevier, vol. 16(2), pages 181-191, October.
    3. Joseph Henrich, 2001. "In Search of Homo Economicus: Behavioral Experiments in 15 Small-Scale Societies," American Economic Review, American Economic Association, vol. 91(2), pages 73-78, May.
    4. Burks, Stephen & Carpenter, Jeffrey & Goette, Lorenz, 2009. "Performance pay and worker cooperation: Evidence from an artefactual field experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 70(3), pages 458-469, June.
    5. Edward L. Glaeser & David I. Laibson & José A. Scheinkman & Christine L. Soutter, 2000. "Measuring Trust," The Quarterly Journal of Economics, Oxford University Press, pages 811-846.
      • Glaeser, Edward Ludwig & Laibson, David I. & Scheinkman, Jose A. & Soutter, Christine L., 2000. "Measuring Trust," Scholarly Articles 4481497, Harvard University Department of Economics.
    6. Roth, Alvin E. & Vesna Prasnikar & Masahiro Okuno-Fujiwara & Shmuel Zamir, 1991. "Bargaining and Market Behavior in Jerusalem, Ljubljana, Pittsburgh, and Tokyo: An Experimental Study," American Economic Review, American Economic Association, vol. 81(5), pages 1068-1095, December.
    7. David J. Cooper, 1999. "Gaming against Managers in Incentive Systems: Experimental Results with Chinese Students and Chinese Managers," American Economic Review, American Economic Association, vol. 89(4), pages 781-804, September.
    8. Armin Falk & James J. Heckman, 2009. "Lab Experiments are a Major Source of Knowledge in the Social Sciences," Working Papers 200935, Geary Institute, University College Dublin.
    9. Baran, Nicole M. & Sapienza, Paola & Zingales, Luigi, 2010. "Can we infer social preferences from the lab? Evidence from the trust game," CEPR Discussion Papers 7634, C.E.P.R. Discussion Papers.
    10. Bellemare, Charles & Kroger, Sabine, 2007. "On representative social capital," European Economic Review, Elsevier, vol. 51(1), pages 183-202, January.
    11. Hans P. Binswanger, 1980. "Attitudes Toward Risk: Experimental Measurement in Rural India," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 62(3), pages 395-407.
    12. Glenn W. Harrison & John A. List, 2004. "Field Experiments," Journal of Economic Literature, American Economic Association, vol. 42(4), pages 1009-1055, December.
    13. Kyriaki Remoundou & Andreas Drichoutis & Phoebe Koundouri, "undated". "Warm glow in charitable auctions: Are the WEIRDos driving the results?," DEOS Working Papers 1026, Athens University of Economics and Business.
    14. R. H. Strotz, 1955. "Myopia and Inconsistency in Dynamic Utility Maximization," Review of Economic Studies, Oxford University Press, vol. 23(3), pages 165-180.
    15. Charles Bellemare & Sabine Kröger & Arthur van Soest, 2008. "Measuring Inequity Aversion in a Heterogeneous Population Using Experimental Decisions and Subjective Probabilities," Econometrica, Econometric Society, vol. 76(4), pages 815-839, July.
    16. Shane Frederick, 2005. "Cognitive Reflection and Decision Making," Journal of Economic Perspectives, American Economic Association, vol. 19(4), pages 25-42, Fall.
    17. John A. List, 2004. "Young, Selfish and Male: Field evidence of social preferences," Economic Journal, Royal Economic Society, vol. 114(492), pages 121-149, January.
    18. John A. List, 2006. "The Behavioralist Meets the Market: Measuring Social Preferences and Reputation Effects in Actual Transactions," Journal of Political Economy, University of Chicago Press, vol. 114(1), pages 1-37, February.
    19. Michèle Belot & Raymond Duch & Luis Miller, 2010. "Who should be called to the lab? A comprehensive comparison of students and non-students in classic experimental games," Discussion Papers 2010001, University of Oxford, Nuffield College.
    20. Harrison, Glenn W. & Lau, Morten I. & Elisabet Rutström, E., 2009. "Risk attitudes, randomization to treatment, and self-selection into experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 70(3), pages 498-507, June.
    21. Steven D. Levitt & John A. List, 2007. "Viewpoint: On the generalizability of lab behaviour to the field," Canadian Journal of Economics, Canadian Economics Association, vol. 40(2), pages 347-370, May.
    22. Carpenter, Jeffrey P. & Burks, Stephen V. & Verhoogen, Eric, 2004. "Comparing Students to Workers: The Effects of Social Framing on Behavior in Distribution Games," IZA Discussion Papers 1341, Institute for the Study of Labor (IZA).
    23. Heckman, James, 2013. "Sample selection bias as a specification error," Applied Econometrics, Publishing House "SINERGIA PRESS", vol. 31(3), pages 129-137.
    24. James Andreoni & Charles Sprenger, 2012. "Estimating Time Preferences from Convex Budgets," American Economic Review, American Economic Association, vol. 102(7), pages 3333-3356, December.
    25. Marco Casari & John C. Ham & John H. Kagel, 2007. "Selection Bias, Demographic Effects, and Ability Effects in Common Value Auction Experiments," American Economic Review, American Economic Association, pages 1278-1304.
    26. Roe, Brian E. & Haab, Timothy C. & Beversdorf, David Q. & Gu, Howard H. & Tilley, Michael R., 2009. "Risk-attitude selection bias in subject pools for experiments involving neuroimaging and blood samples," Journal of Economic Psychology, Elsevier, vol. 30(2), pages 181-189, April.
    27. Werner Güth & Carsten Schmidt & Matthias Sutter, 2007. "Bargaining outside the lab - a newspaper experiment of a three-person ultimatum game," Economic Journal, Royal Economic Society, vol. 117(518), pages 449-469, March.
    28. Binswanger, Hans P, 1981. "Attitudes toward Risk: Theoretical Implications of an Experiment in Rural India," Economic Journal, Royal Economic Society, vol. 91(364), pages 867-890, December.
    29. Jeffrey Carpenter & Cristina Connolly & Caitlin Myers, 2008. "Altruistic behavior in a representative dictator experiment," Experimental Economics, Springer;Economic Science Association, vol. 11(3), pages 282-298, September.
    30. S. Bowles & S. Polania-Reyes., 2013. "Economic Incentives and Social Preferences: Substitutes or Complements?," VOPROSY ECONOMIKI, N.P. Redaktsiya zhurnala "Voprosy Economiki", vol. 4.
    31. Matthias Benz & Stephan Meier, 2008. "Do people behave in experiments as in the field?—evidence from donations," Experimental Economics, Springer;Economic Science Association, vol. 11(3), pages 268-281, September.
    32. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
    33. Garbarino, Ellen & Slonim, Robert, 2009. "The robustness of trust and reciprocity across a heterogeneous U.S. population," Journal of Economic Behavior & Organization, Elsevier, vol. 69(3), pages 226-240, March.
    34. repec:feb:artefa:00073 is not listed on IDEAS
    35. Vivi Alatas & Lisa Cameron & Ananish Chaudhuri & Nisvan Erkal & Lata Gangadharan, 2009. "Subject pool effects in a corruption experiment: A comparison of Indonesian public servants and Indonesian students," Experimental Economics, Springer;Economic Science Association, vol. 12(1), pages 113-132, March.
    36. Andersen, Steffen & Harrison, Glenn W. & Lau, Morten Igel & Rutström, E. Elisabet, 2010. "Preference heterogeneity in experiments: Comparing the field and laboratory," Journal of Economic Behavior & Organization, Elsevier, vol. 73(2), pages 209-224, February.
    37. Omar Al-Ubaydli & John List, 2012. "On the Generalizability of Experimental Results in Economics," Artefactual Field Experiments 00467, The Field Experiments Website.
    38. Catherine C. Eckel & Philip J. Grossman, 2008. "Forecasting Risk Attitudes: An Experimental Study Using Actual and Forecast Gamble Choices," Monash Economics Working Papers archive-01, Monash University, Department of Economics.
    39. Michal Krawczyk, 2011. "What brings your subjects to the lab? A field experiment," Experimental Economics, Springer;Economic Science Association, vol. 14(4), pages 482-489, November.
    40. Cameron, Lisa & Chaudhuri, Ananish & Erkal, Nisvan & Gangadharan, Lata, 2009. "Propensities to engage in and punish corrupt behavior: Experimental evidence from Australia, India, Indonesia and Singapore," Journal of Public Economics, Elsevier, vol. 93(7-8), pages 843-851, August.
    41. Catherine Eckel & Philip Grossman, 2000. "Volunteers and Pseudo-Volunteers: The Effect of Recruitment Method in Dictator Experiments," Experimental Economics, Springer;Economic Science Association, vol. 3(2), pages 107-120, October.
    42. Steven D. Levitt & John A. List, 2007. "What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?," Journal of Economic Perspectives, American Economic Association, vol. 21(2), pages 153-174, Spring.
    43. Ellen Garbarino & Robert Slonim & Justin Sydnor, 2011. "Digit ratios (2D:4D) as predictors of risky decision making for both sexes," Journal of Risk and Uncertainty, Springer, vol. 42(1), pages 1-26, February.
    44. Juan Cardenas, 2011. "Social Norms and Behavior in the Local Commons as Seen Through the Lens of Field Experiments," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 48(3), pages 451-485, March.
    45. Smith, Vernon L, 1976. "Experimental Economics: Induced Value Theory," American Economic Review, American Economic Association, vol. 66(2), pages 274-279, May.
    46. R. Lynn Hannan & John H. Kagel & Donald V. Moser, 2002. "Partial Gift Exchange in an Experimental Labor Market: Impact of Subject Population Differences, Productivity Differences, and Effort Requests on Behavior," Journal of Labor Economics, University of Chicago Press, vol. 20(4), pages 923-951, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Greiner, Ben & Caravella, Mary & Roth, Alvin E., 2014. "Is avatar-to-avatar communication as effective as face-to-face communication? An Ultimatum Game experiment in First and Second Life," Journal of Economic Behavior & Organization, Elsevier, vol. 108(C), pages 374-382.
    2. Matteo M. Galizzi & Daniel Navarro Martinez, 2015. "On the external validity of social-preference games: A systematic lab-field study," Economics Working Papers 1462, Department of Economics and Business, Universitat Pompeu Fabra.
    3. Englmaier, Florian & Gebhardt, Georg, 2016. "Social dilemmas in the laboratory and in the field," Journal of Economic Behavior & Organization, Elsevier, vol. 128(C), pages 85-96.
    4. Alan, Sule & Baydar, Nazli & Boneva, Teodora & Crossley, Thomas F. & Ertac, Seda, 2017. "Transmission of risk preferences from mothers to daughters," Journal of Economic Behavior & Organization, Elsevier, vol. 134(C), pages 60-77.
    5. Galizzi, Matteo M. & Navarro-Martínez, Daniel, 2018. "On the external validity of social preference games: a systematic lab-field study," LSE Research Online Documents on Economics 84088, London School of Economics and Political Science, LSE Library.
    6. Frijters, Paul & Kong, Tao Sherry & Liu, Elaine M., 2015. "Who is coming to the artefactual field experiment? Participation bias among Chinese rural migrants," Journal of Economic Behavior & Organization, Elsevier, vol. 114(C), pages 62-74.
    7. Belot, Michèle & James, Jonathan, 2016. "Partner selection into policy relevant field experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 123(C), pages 31-56.
    8. Kamas, Linda & Preston, Anne, 2016. "Are we underestimating inequality aversion? Comparing recruited and classroom subjects," Economics Letters, Elsevier, vol. 147(C), pages 157-159.
    9. Atalay, Kadir & Bakhtiar, Fayzan & Cheung, Stephen & Slonim, Robert, 2014. "Savings and prize-linked savings accounts," Journal of Economic Behavior & Organization, Elsevier, vol. 107(PA), pages 86-106.
    10. Mirco Tonin & Michael Vlassopoulos, 2015. "Corporate Philanthropy and Productivity: Evidence from an Online Real Effort Experiment," Management Science, INFORMS, vol. 61(8), pages 1795-1811, August.
    11. repec:spr:jesaex:v:3:y:2017:i:2:d:10.1007_s40881-017-0043-0 is not listed on IDEAS
    12. Kettner, Sara Elisa & Waichman, Israel, 2016. "Old age and prosocial behavior: Social preferences or experimental confounds?," Journal of Economic Psychology, Elsevier, vol. 53(C), pages 118-130.
    13. Chivers, David, 2017. "Success, survive or escape? Aspirations and poverty traps," Journal of Economic Behavior & Organization, Elsevier, pages 116-132.
    14. repec:spr:jesaex:v:3:y:2017:i:1:d:10.1007_s40881-017-0036-z is not listed on IDEAS

    More about this item

    Keywords

    Participation bias; Laboratory experiments;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jeborg:v:90:y:2013:i:c:p:43-70. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Dana Niculescu). General contact details of provider: http://www.elsevier.com/locate/jebo .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.