IDEAS home Printed from https://ideas.repec.org/a/spr/grdene/v27y2018i2d10.1007_s10726-018-9565-y.html
   My bibliography  Save this article

Crowd Labor Markets as Platform for Group Decision and Negotiation Research: A Comparison to Laboratory Experiments

Author

Listed:
  • Florian Teschner

    (Karlsruhe Institute of Technology (KIT))

  • Henner Gimpel

    (University of Augsburg)

Abstract

Crowd labor markets such as Amazon Mechanical Turk (MTurk) have emerged as popular platforms where researchers can relatively inexpensively and easily run web-based experiments. Some work even suggests that MTurk can be used to run large-scale field experiments in which groups of participants interact synchronously in real-time such as electronic markets. Besides technical issues, several methodological questions arise and lead to the question of how results from MTurk and laboratory experiments compare. Our data shows comparable results between MTurk and a standard lab setting with student subjects in a controlled environment when running rather simple individual decision tasks. However, our data shows stark differences in results between the experimental settings for a rather complex market experiment. Each experimental setting—lab and MTurk—has its own benefits and drawbacks; which of the two settings is better suited for a specific experiment depends on the theory or artifact to be tested. We discuss potential causes for differences (language understanding, education, cognition and context) that we cannot control for and provide guidance for the selection of the appropriate setting for an experiment. In any case, researchers studying complex artifacts like group decisions or markets should not prematurely adopt MTurk based on extant literature regarding comparable results across experimental settings for rather simple tasks.

Suggested Citation

  • Florian Teschner & Henner Gimpel, 2018. "Crowd Labor Markets as Platform for Group Decision and Negotiation Research: A Comparison to Laboratory Experiments," Group Decision and Negotiation, Springer, vol. 27(2), pages 197-214, April.
    • Handle: RePEc:spr:grdene:v:27:y:2018:i:2:d:10.1007_s10726-018-9565-y
    DOI: 10.1007/s10726-018-9565-y
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10726-018-9565-y
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10726-018-9565-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    2. Mullinix, Kevin J. & Leeper, Thomas J. & Druckman, James N. & Freese, Jeremy, 2015. "The Generalizability of Survey Experiments," Journal of Experimental Political Science, Cambridge University Press, vol. 2(2), pages 109-138, January.
    3. Fair, Ray C & Shiller, Robert J, 1989. "The Informational Context of Ex Ante Forecasts," The Review of Economics and Statistics, MIT Press, vol. 71(2), pages 325-331, May.
    4. Paolacci, Gabriele & Chandler, Jesse & Ipeirotis, Panagiotis G., 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Cambridge University Press, vol. 5(5), pages 411-419, August.
    5. Robin Hanson, 2003. "Combinatorial Information Market Design," Information Systems Frontiers, Springer, vol. 5(1), pages 107-119, January.
    6. Justin Wolfers & Eric Zitzewitz, 2004. "Prediction Markets," Journal of Economic Perspectives, American Economic Association, vol. 18(2), pages 107-126, Spring.
    7. Paul J. Healy & Sera Linardi & J. Richard Lowery & John O. Ledyard, 2010. "Prediction Markets: Alternative Mechanisms for Complex Environments with Few Traders," Management Science, INFORMS, vol. 56(11), pages 1977-1996, November.
    8. Plott, Charles R & Sunder, Shyam, 1988. "Rational Expectations and the Aggregation of Diverse Information in Laboratory Security Markets," Econometrica, Econometric Society, vol. 56(5), pages 1085-1118, September.
    9. Brad M. Barber & Terrance Odean, 2000. "Trading Is Hazardous to Your Wealth: The Common Stock Investment Performance of Individual Investors," Journal of Finance, American Finance Association, vol. 55(2), pages 773-806, April.
    10. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    11. Joyce E. Berg & Thomas A. Rietz, 2003. "Prediction Markets as Decision Support Systems," Information Systems Frontiers, Springer, vol. 5(1), pages 79-93, January.
    12. Ofra Amir & David G Rand & Ya'akov Kobi Gal, 2012. "Economic Games on the Internet: The Effect of $1 Stakes," PLOS ONE, Public Library of Science, vol. 7(2), pages 1-4, February.
    13. Ledyard, John & Hanson, Robin & Ishikida, Takashi, 2009. "An experimental test of combinatorial information markets," Journal of Economic Behavior & Organization, Elsevier, vol. 69(2), pages 182-189, February.
    14. Chen, Daniel L. & Horton, John J., 2016. "Are Online Labor Markets Spot Markets for Tasks?: A Field Experiment on the Behavioral Response to Wage Cuts," TSE Working Papers 16-675, Toulouse School of Economics (TSE).
    15. Jim Lavoie, 2009. "The Innovation Engine at Rite-Solutions: Lessons from the CEO," Journal of Prediction Markets, University of Buckingham Press, vol. 3(1), pages 1-11, April.
    16. Gregory Kersten & Sunil Noronha, 1999. "Negotiation via the World Wide Web: A Cross-cultural Study of Decision Making," Group Decision and Negotiation, Springer, vol. 8(3), pages 251-279, May.
    17. Shane Frederick, 2005. "Cognitive Reflection and Decision Making," Journal of Economic Perspectives, American Economic Association, vol. 19(4), pages 25-42, Fall.
    18. Berg, Joyce E. & Nelson, Forrest D. & Rietz, Thomas A., 2008. "Prediction market accuracy in the long run," International Journal of Forecasting, Elsevier, vol. 24(2), pages 285-300.
    19. Florian Teschner & David Rothschild & Henner Gimpel, 2017. "Manipulation in Conditional Decision Markets," Group Decision and Negotiation, Springer, vol. 26(5), pages 953-971, September.
    20. Tim Straub & Henner Gimpel & Florian Teschner & Christof Weinhardt, 2015. "How (not) to Incent Crowd Workers," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 57(3), pages 167-179, June.
    21. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    22. Bennouri, Moez & Gimpel, Henner & Robert, Jacques, 2011. "Measuring the impact of information aggregation mechanisms: An experimental investigation," Journal of Economic Behavior & Organization, Elsevier, vol. 78(3), pages 302-318, May.
    23. Martin Spann & Bernd Skiera, 2003. "Internet-Based Virtual Stock Markets for Business Forecasting," Management Science, INFORMS, vol. 49(10), pages 1310-1326, October.
    24. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    25. T. S. Breusch & A. R. Pagan, 1980. "The Lagrange Multiplier Test and its Applications to Model Specification in Econometrics," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 47(1), pages 239-253.
    26. Alain Pinsonneault & Henri Barki & R. Brent Gallupe & Norberto Hoppen, 1999. "Electronic Brainstorming: The Illusion of Productivity," Information Systems Research, INFORMS, vol. 10(2), pages 110-133, June.
    27. Chandler, Dana & Kapelner, Adam, 2013. "Breaking monotony with meaning: Motivation in crowdsourcing markets," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 123-133.
    28. Lian Jian & Rahul Sami, 2012. "Aggregation and Manipulation in Prediction Markets: Effects of Trading Mechanism and Information Distribution," Management Science, INFORMS, vol. 58(1), pages 123-140, January.
    29. Antonio Ferreira & Pedro Antunes & Valeria Herskovic, 2011. "Improving Group Attention: An Experiment with Synchronous Brainstorming," Group Decision and Negotiation, Springer, vol. 20(5), pages 643-666, September.
    30. Logan S. Casey & Jesse Chandler & Adam Seth Levine & Andrew Proctor & Dara Z. Strolovitch, 2017. "Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection," SAGE Open, , vol. 7(2), pages 21582440177, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ertel, Christian, 2021. "Die Effekte von Clawback-Klauseln auf das Investitionsverhalten [The Effects of Clawback Provisions on Investment Behaviour]," Junior Management Science (JUMS), Junior Management Science e. V., vol. 6(4), pages 757-789.
    2. Gert Jan Hofstede & Catholijn M. Jonker & Tim Verwaart & Neil Yorke-Smith, 2019. "The Lemon Car Game Across Cultures: Evidence of Relational Rationality," Group Decision and Negotiation, Springer, vol. 28(5), pages 849-877, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Florian Teschner & David Rothschild & Henner Gimpel, 2017. "Manipulation in Conditional Decision Markets," Group Decision and Negotiation, Springer, vol. 26(5), pages 953-971, September.
    2. Lian Jian & Rahul Sami, 2012. "Aggregation and Manipulation in Prediction Markets: Effects of Trading Mechanism and Information Distribution," Management Science, INFORMS, vol. 58(1), pages 123-140, January.
    3. Justin Wolfers & Eric Zitzewitz, 2004. "Prediction Markets," Journal of Economic Perspectives, American Economic Association, vol. 18(2), pages 107-126, Spring.
    4. Spyros Galanis & Christos A Ioannou & Stelios Kotronis, 2024. "Information Aggregation Under Ambiguity: Theory and Experimental Evidence," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 91(6), pages 3423-3467.
    5. Prissé, Benjamin & Jorrat, Diego, 2022. "Lab vs online experiments: No differences," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    6. Ledyard, John & Hanson, Robin & Ishikida, Takashi, 2009. "An experimental test of combinatorial information markets," Journal of Economic Behavior & Organization, Elsevier, vol. 69(2), pages 182-189, February.
    7. Snowberg, Erik & Wolfers, Justin & Zitzewitz, Eric, 2013. "Prediction Markets for Economic Forecasting," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 657-687, Elsevier.
    8. Forsell, Eskil & Viganola, Domenico & Pfeiffer, Thomas & Almenberg, Johan & Wilson, Brad & Chen, Yiling & Nosek, Brian A. & Johannesson, Magnus & Dreber, Anna, 2019. "Predicting replication outcomes in the Many Labs 2 study," Journal of Economic Psychology, Elsevier, vol. 75(PA).
    9. Page, Lionel & Siemroth, Christoph, 2017. "An experimental analysis of information acquisition in prediction markets," Games and Economic Behavior, Elsevier, vol. 101(C), pages 354-378.
    10. Edoardo Gaffeo, 2013. "Using information markets in grantmaking. An assessment of the issues involved and an application to Italian banking foundations," DEM Discussion Papers 2013/08, Department of Economics and Management.
    11. Antonio A. Arechar & Gordon T. Kraft-Todd & David G. Rand, 2017. "Turking overtime: how participant characteristics and behavior vary over time and day on Amazon Mechanical Turk," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 3(1), pages 1-11, July.
    12. Schnitzlein, Charles & Chelley-Steeley, Patricia & Steeley, James M, 2024. "Conflicting versus reinforcing private information, information aggregation, and the time series properties of asset prices," Journal of Banking & Finance, Elsevier, vol. 169(C).
    13. Brañas-Garza, Pablo & Capraro, Valerio & Rascón-Ramírez, Ericka, 2018. "Gender differences in altruism on Mechanical Turk: Expectations and actual behaviour," Economics Letters, Elsevier, vol. 170(C), pages 19-23.
    14. Lawrence Choo & Todd R. Kaplan & Ro’i Zultan, 2022. "Manipulation and (Mis)trust in Prediction Markets," Management Science, INFORMS, vol. 68(9), pages 6716-6732, September.
    15. Karimi, Majid & Zaerpour, Nima, 2022. "Put your money where your forecast is: Supply chain collaborative forecasting with cost-function-based prediction markets," European Journal of Operational Research, Elsevier, vol. 300(3), pages 1035-1049.
    16. Ahrash Dianat & Christoph Siemroth, 2021. "Improving decisions with market information: an experiment on corporate prediction markets," Experimental Economics, Springer;Economic Science Association, vol. 24(1), pages 143-176, March.
    17. Van Borm, Hannah & Burn, Ian & Baert, Stijn, 2021. "What Does a Job Candidate's Age Signal to Employers?," Labour Economics, Elsevier, vol. 71(C).
    18. Patrick Buckley & Fergal O’Brien, 0. "The effect of malicious manipulations on prediction market accuracy," Information Systems Frontiers, Springer, vol. 0, pages 1-13.
    19. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    20. Eriksen, Kristoffer W. & Fest, Sebastian & Kvaløy, Ola & Dijk, Oege, 2022. "Fair advice," Journal of Banking & Finance, Elsevier, vol. 143(C).

    More about this item

    Keywords

    Experiments; Mechanical turk; Electronic markets; Information aggregation;
    All these keywords.

    JEL classification:

    • C9 - Mathematical and Quantitative Methods - - Design of Experiments
    • D8 - Microeconomics - - Information, Knowledge, and Uncertainty

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:grdene:v:27:y:2018:i:2:d:10.1007_s10726-018-9565-y. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.