IDEAS home Printed from https://ideas.repec.org/p/swe/wpaper/2015-20a.html
   My bibliography  Save this paper

MTurk ‘Unscrubbed’: Exploring the good, the ‘Super’, and the unreliable on Amazon’s Mechanical Turk

Author

Listed:
  • Jeanette A.M.J. Deetlefs

    (School of Marketing, UNSW Business School, UNSW)

  • Mathew Chylinski

    (School of Marketing, UNSW Business School, UNSW)

  • Andreas Ortmann

    (School of Economics, UNSW Business School, UNSW)

Abstract

Widely accepted as a low-cost, fast-turnaround solution with acceptable validity, Amazon’s Mechanical Turk (MTurk) is increasingly being used to source participants for academic studies (Berinsky et al. 2012; Bohannon 2011; Chandler et al. 2014; Mason and Suri 2012). Yet two commonly raised concerns remain: the presence of quasi-professional respondents, or “Super-Turkers”, and the presence of “Spammers”, those that compromise quality while optimising their pay rate. We isolate the influence on research results of experienced subjects (Super-Turkers), and of unreliable subjects (Spammers), jointly and separately. Jointly including these subjects produces very similar results to jointly excluding them, yet effect sizes decrease disproportionately to their sample representation. Furthermore, separately including experienced subjects in research results is shown to be as problematic as inclusion of unreliable subjects, although the noise introduced by these subjects is divergent and measure dependent. Hence removing only one of these types of respondents can be even more damaging to the reliability of results, than including both.

Suggested Citation

  • Jeanette A.M.J. Deetlefs & Mathew Chylinski & Andreas Ortmann, 2015. "MTurk ‘Unscrubbed’: Exploring the good, the ‘Super’, and the unreliable on Amazon’s Mechanical Turk," Discussion Papers 2015-20, School of Economics, The University of New South Wales.
    • Handle: RePEc:swe:wpaper:2015-20a
    as

    Download full text from publisher

    File URL: http://research.economics.unsw.edu.au/RePEc/papers/2015-20.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    2. Camerer, Colin F & Hogarth, Robin M, 1999. "The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework," Journal of Risk and Uncertainty, Springer, vol. 19(1-3), pages 7-42, December.
    3. van Rooij, Maarten & Lusardi, Annamaria & Alessie, Rob, 2011. "Financial literacy and stock market participation," Journal of Financial Economics, Elsevier, vol. 101(2), pages 449-472, August.
    4. Todd L. Cherry & Peter Frykblom & Jason F. Shogren, 2002. "Hardnose the Dictator," American Economic Review, American Economic Association, vol. 92(4), pages 1218-1221, September.
    5. Lusardi, Annamaria & Mitchell, Olivia S., 2007. "Baby Boomer retirement security: The roles of planning, financial literacy, and housing wealth," Journal of Monetary Economics, Elsevier, vol. 54(1), pages 205-224, January.
    6. Hazel Bateman & Christine Eckert & John Geweke & Jordan Louviere & Susan Thorp & Stephen Satchell, 2012. "Financial Competence and Expectations Formation: Evidence from Australia," The Economic Record, The Economic Society of Australia, vol. 88(280), pages 39-63, March.
    7. Annamaria Lusardi & Olivia S. Mitchell, 2017. "How Ordinary Consumers Make Complex Economic Decisions: Financial Literacy and Retirement Readiness," Quarterly Journal of Finance (QJF), World Scientific Publishing Co. Pte. Ltd., vol. 7(03), pages 1-31, September.
    8. Uri Gneezy & Jan Potters, 1997. "An Experiment on Risk Taking and Evaluation Periods," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 112(2), pages 631-645.
    9. Catherine C. Eckel & Philip J. Grossman, 2008. "Forecasting Risk Attitudes: An Experimental Study Using Actual and Forecast Gamble Choices," Monash Economics Working Papers archive-01, Monash University, Department of Economics.
    10. Charness, Gary & Gneezy, Uri & Imas, Alex, 2013. "Experimental methods: Eliciting risk preferences," Journal of Economic Behavior & Organization, Elsevier, vol. 87(C), pages 43-51.
    11. Charness, Gary & Karni, Edi & Levin, Dan, 2010. "On the conjunction fallacy in probability judgment: New experimental evidence regarding Linda," Games and Economic Behavior, Elsevier, vol. 68(2), pages 551-556, March.
    12. Kaufman, Carol Felker & Lane, Paul M & Lindquist, Jay D, 1991. "Exploring More Than 24 Hours a Day: A Preliminary Investigation of Polychronic Time Use," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 18(3), pages 392-401, December.
    13. Reilly, Michael D, 1982. "Working Wives and Convenience Consumption," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 8(4), pages 407-418, March.
    14. David G. Rand & Alexander Peysakhovich & Gordon T. Kraft-Todd & George E. Newman & Owen Wurzbacher & Martin A. Nowak & Joshua D. Greene, 2014. "Social heuristics shape intuitive cooperation," Nature Communications, Nature, vol. 5(1), pages 1-12, May.
    15. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Antonio A. Arechar & Gordon T. Kraft-Todd & David G. Rand, 2017. "Turking overtime: how participant characteristics and behavior vary over time and day on Amazon Mechanical Turk," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 3(1), pages 1-11, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jeanette A.M.J. Deetlefs & Mathew Chylinski & Andreas Ortmann, 2015. "MTurk ‘Unscrubbed’: Exploring the good, the ‘Super’, and the unreliable on Amazon’s Mechanical Turk," Discussion Papers 2015-20, School of Economics, The University of New South Wales.
    2. Wladislaw Mill & Cornelius Schneider, 2023. "The Bright Side of Tax Evasion," CESifo Working Paper Series 10615, CESifo.
    3. Ranganathan, Kavitha & Lejarraga, Tomás, 2021. "Elicitation of risk preferences through satisficing," Journal of Behavioral and Experimental Finance, Elsevier, vol. 32(C).
    4. Prissé, Benjamin & Jorrat, Diego, 2022. "Lab vs online experiments: No differences," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    5. repec:cup:judgdm:v:16:y:2021:i:5:p:1290-1313 is not listed on IDEAS
    6. Collier, Trevor & Cotten, Stephen & Roush, Justin, 2022. "Using pandemic behavior to test the external validity of laboratory measurements of risk aversion and guilt," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 101(C).
    7. Christian Ehm & Christine Laudenbach & Martin Weber, 2018. "Focusing on volatility information instead of portfolio weights as an aid to investor decisions," Experimental Economics, Springer;Economic Science Association, vol. 21(2), pages 457-480, June.
    8. Noviarini, Jelita & Coleman, Andrew & Roberts, Helen & Whiting, Rosalind H., 2023. "Financial literacy and retirees' resource allocation decisions in New Zealand," Pacific-Basin Finance Journal, Elsevier, vol. 79(C).
    9. Brañas-Garza, Pablo & Estepa-Mohedano, Lorenzo & Jorrat, Diego & Orozco, Victor & Rascón-Ramírez, Ericka, 2021. "To pay or not to pay: Measuring risk preferences in lab and field," Judgment and Decision Making, Cambridge University Press, vol. 16(5), pages 1290-1313, September.
    10. Lusardi, Annamaria & Mitchell, Olivia S., 2011. "Financial literacy around the world: an overview," Journal of Pension Economics and Finance, Cambridge University Press, vol. 10(4), pages 497-508, October.
    11. repec:cup:judgdm:v:14:y:2019:i:3:p:234-279 is not listed on IDEAS
    12. Lefgren, Lars J. & Sims, David P. & Stoddard, Olga B., 2016. "Effort, luck, and voting for redistribution," Journal of Public Economics, Elsevier, vol. 143(C), pages 89-97.
    13. Joanne W. Hsu, 2016. "Aging and Strategic Learning: The Impact of Spousal Incentives on Financial Literacy," Journal of Human Resources, University of Wisconsin Press, vol. 51(4), pages 1036-1067.
    14. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell us about p-Hacking and Publication Bias in Online Experiments," GLO Discussion Paper Series 1157, Global Labor Organization (GLO).
    15. Shin, Su Hyun & Kim, Kyoung Tae & Seay, Martin, 2020. "Sources of information and portfolio allocation," Journal of Economic Psychology, Elsevier, vol. 76(C).
    16. Sholevar, Maryam & Harris, Laurence, 2019. "Mind the gap: A discussion paper on Financial Literacy, Financial behaviour and Financial Education : Is there any Gender Gap?," OSF Preprints b7zd6, Center for Open Science.
    17. Oscar A. Stolper & Andreas Walter, 2017. "Financial literacy, financial advice, and financial behavior," Journal of Business Economics, Springer, vol. 87(5), pages 581-643, July.
    18. Baeckström, Ylva & Marsh, Ian W. & Silvester, Joanne, 2021. "Variations in investment advice provision: A study of financial advisors of millionaire investors," Journal of Economic Behavior & Organization, Elsevier, vol. 188(C), pages 716-735.
    19. Annamaria Lusardi & Olivia S. Mitchell & Vilsa Curto, 2009. "Financial Literacy and Financial Sophistication Among Older Americans," NBER Working Papers 15469, National Bureau of Economic Research, Inc.
    20. Caliendo, Frank N. & Findley, T. Scott, 2013. "Time inconsistency and retirement planning," Economics Letters, Elsevier, vol. 121(1), pages 30-34.
    21. Di Girolamo, Amalia & Harrison, Glenn W. & Lau, Morten I. & Swarthout, J. Todd, 2015. "Subjective belief distributions and the characterization of economic literacy," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 59(C), pages 1-12.
    22. Sascha Füllbrunn & Wolfgang J. Luhan, 2015. "Am I my Peer‘s Keeper? Social Responsibility in Financial Decision Making," Ruhr Economic Papers 0551, Rheinisch-Westfälisches Institut für Wirtschaftsforschung, Ruhr-Universität Bochum, Universität Dortmund, Universität Duisburg-Essen.

    More about this item

    Keywords

    data collection; experimentation; field experiment; internet; Mechanical Turk;
    All these keywords.

    JEL classification:

    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • D80 - Microeconomics - - Information, Knowledge, and Uncertainty - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:swe:wpaper:2015-20a. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Hongyi Li (email available below). General contact details of provider: https://edirc.repec.org/data/senswau.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.