Running experiments on Amazon Mechanical Turk
AbstractAlthough Mechanical Turk has recently become popular among social scientists as a source of experimental data, doubts may linger about the quality of data provided by subjects recruited from online labor markets. We address these potential concerns by presenting new demographic data about the Mechanical Turk subject population, reviewing the strengths of Mechanical Turk relative to other online and offline methods of recruiting subjects, and comparing the magnitude of effects obtained using Mechanical Turk and traditional subject pools. We further discuss some additional benefits such as the possibility of longitudinal, cross cultural and prescreening designs, and offer some advice on how to best manage a common subject pool.
Download InfoIf you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
Bibliographic InfoArticle provided by Society for Judgment and Decision Making in its journal Judgment and Decision Making.
Volume (Year): 5 (2010)
Issue (Month): 5 (August)
Contact details of provider:
experimentation; online research;
You can help add them by filling out this form.
CitEc Project, subscribe to its RSS feed for this item.
- Katherine B. Coffman & Lucas C. Coffman & Keith M. Marzilli Ericson, 2013. "The Size of the LGBT Population and the Magnitude of Anti-Gay Sentiment are Substantially Underestimated," NBER Working Papers 19508, National Bureau of Economic Research, Inc.
- Aguinis, Herman & Lawal, Sola O., 2012. "Conducting field experiments using eLancing's natural environment," Journal of Business Venturing, Elsevier, vol. 27(4), pages 493-505.
- Yury Shevchenko & Bettina von Helversen & Benjamin Scheibehenne, 2014. "Change and status quo in decisions with defaults: The effect of incidental emotions depends on the type of default," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 9(3), pages 287-296, May.
- Yamada, Katsunori & Sato, Masayuki, 2013.
"Another avenue for anatomy of income comparisons: Evidence from hypothetical choice experiments,"
Journal of Economic Behavior & Organization,
Elsevier, vol. 89(C), pages 35-57.
- Katsunori Yamada & Masayuki Sato, 2010. "Another Avenue for Anatomy of Income Comparisons: Evidence from Hypothetical Choice Experiments," ISER Discussion Paper 0795, Institute of Social and Economic Research, Osaka University, revised Mar 2013.
- Wieland, Alice & Sarin, Rakesh, 2012. "Domain specificity of sex differences in competition," Journal of Economic Behavior & Organization, Elsevier, vol. 83(1), pages 151-157.
- Manapat, Michael L. & Nowak, Martin A. & Rand, David G., 2013. "Information, irrationality, and the evolution of trust," Journal of Economic Behavior & Organization, Elsevier, vol. 90(S), pages S57-S75.
- Polman, Evan, 2012. "Self–other decision making and loss aversion," Organizational Behavior and Human Decision Processes, Elsevier, vol. 119(2), pages 141-150.
- Dreber Almenberg, Anna & Gerdes, Christer & Gränsmark, Patrik, 2010.
"Beauty Queens and Battling Knights: Risk Taking and Attractiveness in Chess,"
IZA Discussion Papers
5314, Institute for the Study of Labor (IZA).
- Dreber, Anna & Gerdes, Christer & Gränsmark, Patrik, 2013. "Beauty queens and battling knights: Risk taking and attractiveness in chess," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 1-18.
- Gupta, Vishal K. & Goktan, A. Banu & Gunay, Gonca, 2014. "Gender differences in evaluation of new business opportunity: A stereotype threat perspective," Journal of Business Venturing, Elsevier, vol. 29(2), pages 273-288.
- Boyer O’Leary, Michael & Wilson, Jeanne M. & Metiu, Anca, 2011.
"Beyond Being There: The Symbolic Role of Communication and Identification in the Emergence of Perceived Proximity in Geographically Dispersed Work,"
ESSEC Working Papers
WP1112, ESSEC Research Center, ESSEC Business School.
- Michael Boyer O'leary & Anca Metiu & Jeanne M. Wilson, 2011. "Beyond Being There: The Symbolic Role of Communication and Identification in the Emergence of Perceived Proximity in Geographically Dispersed Work," Post-Print hal-00661000, HAL.
- Charness, Gary & Gneezy, Uri & Kuhn, Michael A., 2013. "Experimental methods: Extra-laboratory experiments-extending the reach of experimental economics," Journal of Economic Behavior & Organization, Elsevier, vol. 91(C), pages 93-100.
- Chandler, Dana & Kapelner, Adam, 2013. "Breaking monotony with meaning: Motivation in crowdsourcing markets," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 123-133.
- Sharma, Eesha & Mazar, Nina & Alter, Adam L. & Ariely, Dan, 2014. "Financial deprivation selectively shifts moral standards and compromises moral decisions," Organizational Behavior and Human Decision Processes, Elsevier, vol. 123(2), pages 90-100.
- Armenak Antinyan, 2014. "Loss and Other-Regarding Preferences: Evidence From Dictator Game," Working Papers 3, Department of Management, Università Ca' Foscari Venezia.
- David Cooper & David Johnson, 2013. "Ambiguity in Performance Pay: An Online Experiment," Working Papers 2013-27, Department of Economics, University of Calgary, revised 14 Nov 2013.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Jonathan Baron).
If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.
If references are entirely missing, you can add them using this form.
If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.
If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.
Please note that corrections may take a couple of weeks to filter through the various RePEc services.