IDEAS home Printed from https://ideas.repec.org/a/spr/jesaex/v3y2017i1d10.1007_s40881-017-0035-0.html
   My bibliography  Save this article

Turking overtime: how participant characteristics and behavior vary over time and day on Amazon Mechanical Turk

Author

Listed:
  • Antonio A. Arechar

    (Yale University)

  • Gordon T. Kraft-Todd

    (Yale University)

  • David G. Rand

    (Yale University
    Yale University
    Yale University)

Abstract

Online experiments allow researchers to collect datasets at times not typical of laboratory studies. We recruit 2336 participants from Amazon Mechanical Turk to examine if participant characteristics and behaviors differ depending on whether the experiment is conducted during the day versus night, and on weekdays versus weekends. Participants make incentivized decisions involving prosociality, punishment, and discounting, and complete a demographic and personality survey. We find no time or day differences in behavior, but do find that participants at nights and on weekends are less experienced with online studies; on weekends are less reflective; and at night are less conscientious and more neurotic. These results are largely robust to finer-grained measures of time and day. We also find that those who participated earlier in the course of the study are more experienced, reflective, and agreeable, but less charitable than later participants.

Suggested Citation

  • Antonio A. Arechar & Gordon T. Kraft-Todd & David G. Rand, 2017. "Turking overtime: how participant characteristics and behavior vary over time and day on Amazon Mechanical Turk," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 3(1), pages 1-11, July.
    • Handle: RePEc:spr:jesaex:v:3:y:2017:i:1:d:10.1007_s40881-017-0035-0
    DOI: 10.1007/s40881-017-0035-0
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s40881-017-0035-0
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s40881-017-0035-0?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    2. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    3. Jesse Chandler & Gabriele Paolacci & Eyal Peer & Pam Mueller & Kate A. Ratliff, 2015. "Using Nonnaive Participants Can Reduce Effect Sizes," Mathematica Policy Research Reports bffac982a56e4cfba3659e74a, Mathematica Policy Research.
    4. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    5. Jeanette A.M.J. Deetlefs & Mathew Chylinski & Andreas Ortmann, 2015. "MTurk ‘Unscrubbed’: Exploring the good, the ‘Super’, and the unreliable on Amazon’s Mechanical Turk," Discussion Papers 2015-20, School of Economics, The University of New South Wales.
    6. Ofra Amir & David G Rand & Ya'akov Kobi Gal, 2012. "Economic Games on the Internet: The Effect of $1 Stakes," PLOS ONE, Public Library of Science, vol. 7(2), pages 1-4, February.
    7. Shane Frederick, 2005. "Cognitive Reflection and Decision Making," Journal of Economic Perspectives, American Economic Association, vol. 19(4), pages 25-42, Fall.
    8. Jeanette A.M.J. Deetlefs & Mathew Chylinski & Andreas Ortmann, 2015. "MTurk ‘Unscrubbed’: Exploring the good, the ‘Super’, and the unreliable on Amazon’s Mechanical Turk," Discussion Papers 2015-20, School of Economics, The University of New South Wales.
    9. David G. Rand & Alexander Peysakhovich & Gordon T. Kraft-Todd & George E. Newman & Owen Wurzbacher & Martin A. Nowak & Joshua D. Greene, 2014. "Social heuristics shape intuitive cooperation," Nature Communications, Nature, vol. 5(1), pages 1-12, May.
    10. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Gary Bolton & Eugen Dimant & Ulrich Schmidt, 2018. "When a Nudge Backfires. Using Observation with Social and Economic Incentives to Promote Pro-Social Behavior," PPE Working Papers 0017, Philosophy, Politics and Economics, University of Pennsylvania.
    2. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell us about p-Hacking and Publication Bias in Online Experiments," GLO Discussion Paper Series 1157, Global Labor Organization (GLO).
    3. Keigo Inukai & Yuta Shimodaira & Kohei Shiozawa, 2022. "Empirical properties of an extended CES utility function in representing distributional preferences," ISER Discussion Paper 1199, Institute of Social and Economic Research, Osaka University.
    4. David Johnson & John Barry Ryan, 2020. "Amazon Mechanical Turk workers can provide consistent and economically meaningful data," Southern Economic Journal, John Wiley & Sons, vol. 87(1), pages 369-385, July.
    5. Arechar, Antonio A. & Rand, David G., 2022. "Learning to be selfish? A large-scale longitudinal analysis of Dictator games played on Amazon Mechanical Turk," Journal of Economic Psychology, Elsevier, vol. 90(C).
    6. Abel Brodeur, Nikolai M. Cook, Anthony Heyes, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell Us about Publication Bias and p-Hacking in Online Experiments," LCERPA Working Papers am0133, Laurier Centre for Economic Research and Policy Analysis.
    7. Gary Charness & David J. Cooper & Zachary Grossman, 2020. "Silence is golden: team problem solving and communication costs," Experimental Economics, Springer;Economic Science Association, vol. 23(3), pages 668-693, September.
    8. Cristina Bicchieri & Eugen Dimant, 2018. "It's Not A Lie If You Believe It. Lying and Belief Distortion Under Norm-Uncertainty," PPE Working Papers 0012, Philosophy, Politics and Economics, University of Pennsylvania.
    9. Binder, Carola Conces, 2022. "Time-of-day and day-of-week variations in Amazon Mechanical Turk survey responses," Journal of Macroeconomics, Elsevier, vol. 71(C).
    10. Chapkovski, Philipp, 2023. "Conducting interactive experiments on Toloka," Journal of Behavioral and Experimental Finance, Elsevier, vol. 37(C).
    11. repec:cup:judgdm:v:13:y:2018:i:3:p:260-267 is not listed on IDEAS
    12. Michael N. Stagnaro & Gordon Pennycook & David G. Rand, 2018. "Performance on the Cognitive Reflection Test is stable across time," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 13(3), pages 260-267, May.
    13. Cherry, Todd L. & James, Alexander G. & Murphy, James, 2021. "The impact of public health messaging and personal experience on the acceptance of mask wearing during the COVID-19 pandemic," Journal of Economic Behavior & Organization, Elsevier, vol. 187(C), pages 415-430.
    14. Fehr, Dietmar & Vollmann, Martin, 2020. "Misperceiving Economic Success: Experimental Evidence on Meritocratic Beliefs and Inequality Acceptance," Working Papers 0695, University of Heidelberg, Department of Economics.
    15. R. Gordon Rinderknecht, 2019. "Effects of Participant Displeasure on the Social-Psychological Study of Power on Amazon’s Mechanical Turk," SAGE Open, , vol. 9(3), pages 21582440198, September.
    16. Chapkovski, Philipp, 2022. "Interactive experiments in Toloka," MPRA Paper 111980, University Library of Munich, Germany.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Prissé, Benjamin & Jorrat, Diego, 2022. "Lab vs online experiments: No differences," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    2. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    3. Wladislaw Mill & Cornelius Schneider, 2023. "The Bright Side of Tax Evasion," CESifo Working Paper Series 10615, CESifo.
    4. Brañas-Garza, Pablo & Capraro, Valerio & Rascón-Ramírez, Ericka, 2018. "Gender differences in altruism on Mechanical Turk: Expectations and actual behaviour," Economics Letters, Elsevier, vol. 170(C), pages 19-23.
    5. Johannes G. Jaspersen & Marc A. Ragin & Justin R. Sydnor, 2022. "Insurance demand experiments: Comparing crowdworking to the lab," Journal of Risk & Insurance, The American Risk and Insurance Association, vol. 89(4), pages 1077-1107, December.
    6. David G Rand & Gordon Kraft-Todd & June Gruber, 2015. "The Collective Benefits of Feeling Good and Letting Go: Positive Emotion and (dis)Inhibition Interact to Predict Cooperative Behavior," PLOS ONE, Public Library of Science, vol. 10(1), pages 1-12, January.
    7. Wladislaw Mill & Jonathan Staebler, 2023. "Spite in Litigation," CRC TR 224 Discussion Paper Series crctr224_2023_401, University of Bonn and University of Mannheim, Germany.
    8. Irene Scopelliti & H. Lauren Min & Erin McCormick & Karim S. Kassam & Carey K. Morewedge, 2018. "Individual Differences in Correspondence Bias: Measurement, Consequences, and Correction of Biased Interpersonal Attributions," Management Science, INFORMS, vol. 64(4), pages 1879-1910, April.
    9. repec:cup:judgdm:v:11:y:2016:i:1:p:99-113 is not listed on IDEAS
    10. Keela S. Thomson & Daniel M. Oppenheimer, 2016. "Investigating an alternate form of the cognitive reflection test," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 11(1), pages 99-113, January.
    11. Capraro, Valerio & Schulz, Jonathan & Rand, David G., 2019. "Time pressure and honesty in a deception game," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 79(C), pages 93-99.
    12. Alempaki, Despoina & Starmer, Chris & Tufano, Fabio, 2019. "On the priming of risk preferences: The role of fear and general affect," Journal of Economic Psychology, Elsevier, vol. 75(PA).
    13. Prissé, Benjamin & Jorrat, Diego, 2021. "Lack of Control: An experiment," MPRA Paper 109918, University Library of Munich, Germany.
    14. Jeremy Cone & David G Rand, 2014. "Time Pressure Increases Cooperation in Competitively Framed Social Dilemmas," PLOS ONE, Public Library of Science, vol. 9(12), pages 1-13, December.
    15. Florian Teschner & Henner Gimpel, 2018. "Crowd Labor Markets as Platform for Group Decision and Negotiation Research: A Comparison to Laboratory Experiments," Group Decision and Negotiation, Springer, vol. 27(2), pages 197-214, April.
    16. Jeanette A.M.J. Deetlefs & Mathew Chylinski & Andreas Ortmann, 2015. "MTurk ‘Unscrubbed’: Exploring the good, the ‘Super’, and the unreliable on Amazon’s Mechanical Turk," Discussion Papers 2015-20, School of Economics, The University of New South Wales.
    17. Dato, Simon & Feess, Eberhard & Nieken, Petra, 2019. "Lying and reciprocity," Games and Economic Behavior, Elsevier, vol. 118(C), pages 193-218.
    18. Gandullia, Luca & Lezzi, Emanuela & Parciasepe, Paolo, 2020. "Replication with MTurk of the experimental design by Gangadharan, Grossman, Jones & Leister (2018): Charitable giving across donor types," Journal of Economic Psychology, Elsevier, vol. 78(C).
    19. Pablo Brañas-Garza & Diego Jorrat & Antonio M. Espín & Angel Sánchez, 2023. "Paid and hypothetical time preferences are the same: lab, field and online evidence," Experimental Economics, Springer;Economic Science Association, vol. 26(2), pages 412-434, April.
    20. Ola Andersson & Jim Ingebretsen Carlson & Erik Wengström, 2021. "Differences Attract: An Experimental Study of Focusing in Economic Choice," The Economic Journal, Royal Economic Society, vol. 131(639), pages 2671-2692.
    21. Nicolas Jacquemet & Alexander G James & Stéphane Luchini & James J Murphy & Jason F Shogren, 2021. "Do truth-telling oaths improve honesty in crowd-working?," PLOS ONE, Public Library of Science, vol. 16(1), pages 1-18, January.

    More about this item

    Keywords

    Cooperation; Honesty; Decision-making; Time of day; MTurk; Self-control;
    All these keywords.

    JEL classification:

    • C80 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - General
    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:jesaex:v:3:y:2017:i:1:d:10.1007_s40881-017-0035-0. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.