IDEAS home Printed from
   My bibliography  Save this article

Crowd Labor Markets as Platform for Group Decision and Negotiation Research: A Comparison to Laboratory Experiments


  • Florian Teschner

    () (Karlsruhe Institute of Technology (KIT))

  • Henner Gimpel

    () (University of Augsburg)


Abstract Crowd labor markets such as Amazon Mechanical Turk (MTurk) have emerged as popular platforms where researchers can relatively inexpensively and easily run web-based experiments. Some work even suggests that MTurk can be used to run large-scale field experiments in which groups of participants interact synchronously in real-time such as electronic markets. Besides technical issues, several methodological questions arise and lead to the question of how results from MTurk and laboratory experiments compare. Our data shows comparable results between MTurk and a standard lab setting with student subjects in a controlled environment when running rather simple individual decision tasks. However, our data shows stark differences in results between the experimental settings for a rather complex market experiment. Each experimental setting—lab and MTurk—has its own benefits and drawbacks; which of the two settings is better suited for a specific experiment depends on the theory or artifact to be tested. We discuss potential causes for differences (language understanding, education, cognition and context) that we cannot control for and provide guidance for the selection of the appropriate setting for an experiment. In any case, researchers studying complex artifacts like group decisions or markets should not prematurely adopt MTurk based on extant literature regarding comparable results across experimental settings for rather simple tasks.

Suggested Citation

  • Florian Teschner & Henner Gimpel, 2018. "Crowd Labor Markets as Platform for Group Decision and Negotiation Research: A Comparison to Laboratory Experiments," Group Decision and Negotiation, Springer, vol. 27(2), pages 197-214, April.
  • Handle: RePEc:spr:grdene:v:27:y:2018:i:2:d:10.1007_s10726-018-9565-y
    DOI: 10.1007/s10726-018-9565-y

    Download full text from publisher

    File URL:
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    1. Mullinix, Kevin J. & Leeper, Thomas J. & Druckman, James N. & Freese, Jeremy, 2015. "The Generalizability of Survey Experiments," Journal of Experimental Political Science, Cambridge University Press, vol. 2(2), pages 109-138, January.
    2. Fair, Ray C & Shiller, Robert J, 1989. "The Informational Context of Ex Ante Forecasts," The Review of Economics and Statistics, MIT Press, vol. 71(2), pages 325-331, May.
    3. Plott, Charles R & Sunder, Shyam, 1988. "Rational Expectations and the Aggregation of Diverse Information in Laboratory Security Markets," Econometrica, Econometric Society, vol. 56(5), pages 1085-1118, September.
    4. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    5. Martin Spann & Bernd Skiera, 2003. "Internet-Based Virtual Stock Markets for Business Forecasting," Management Science, INFORMS, vol. 49(10), pages 1310-1326, October.
    6. T. S. Breusch & A. R. Pagan, 1980. "The Lagrange Multiplier Test and its Applications to Model Specification in Econometrics," Review of Economic Studies, Oxford University Press, vol. 47(1), pages 239-253.
    7. Alain Pinsonneault & Henri Barki & R. Brent Gallupe & Norberto Hoppen, 1999. "Electronic Brainstorming: The Illusion of Productivity," Information Systems Research, INFORMS, vol. 10(2), pages 110-133, June.
    8. Antonio Ferreira & Pedro Antunes & Valeria Herskovic, 2011. "Improving Group Attention: An Experiment with Synchronous Brainstorming," Group Decision and Negotiation, Springer, vol. 20(5), pages 643-666, September.
    9. Robin Hanson, 2003. "Combinatorial Information Market Design," Information Systems Frontiers, Springer, vol. 5(1), pages 107-119, January.
    10. Brad M. Barber & Terrance Odean, 2000. "Trading Is Hazardous to Your Wealth: The Common Stock Investment Performance of Individual Investors," Journal of Finance, American Finance Association, vol. 55(2), pages 773-806, April.
    11. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    12. Joyce E. Berg & Thomas A. Rietz, 2003. "Prediction Markets as Decision Support Systems," Information Systems Frontiers, Springer, vol. 5(1), pages 79-93, January.
    13. Gregory Kersten & Sunil Noronha, 1999. "Negotiation via the World Wide Web: A Cross-cultural Study of Decision Making," Group Decision and Negotiation, Springer, vol. 8(3), pages 251-279, May.
    14. Bennouri, Moez & Gimpel, Henner & Robert, Jacques, 2011. "Measuring the impact of information aggregation mechanisms: An experimental investigation," Journal of Economic Behavior & Organization, Elsevier, vol. 78(3), pages 302-318, May.
    15. Paul J. Healy & Sera Linardi & J. Richard Lowery & John O. Ledyard, 2010. "Prediction Markets: Alternative Mechanisms for Complex Environments with Few Traders," Management Science, INFORMS, vol. 56(11), pages 1977-1996, November.
    16. Chen, Daniel L. & Horton, John J., 2016. "Are Online Labor Markets Spot Markets for Tasks?: A Field Experiment on the Behavioral Response to Wage Cuts," TSE Working Papers 16-675, Toulouse School of Economics (TSE).
    17. Shane Frederick, 2005. "Cognitive Reflection and Decision Making," Journal of Economic Perspectives, American Economic Association, vol. 19(4), pages 25-42, Fall.
    18. Florian Teschner & David Rothschild & Henner Gimpel, 2017. "Manipulation in Conditional Decision Markets," Group Decision and Negotiation, Springer, vol. 26(5), pages 953-971, September.
    19. Ledyard, John & Hanson, Robin & Ishikida, Takashi, 2009. "An experimental test of combinatorial information markets," Journal of Economic Behavior & Organization, Elsevier, vol. 69(2), pages 182-189, February.
    20. Jim Lavoie, 2009. "The Innovation Engine at Rite-Solutions: Lessons from the CEO," Journal of Prediction Markets, University of Buckingham Press, vol. 3(1), pages 1-11, April.
    21. Berg, Joyce E. & Nelson, Forrest D. & Rietz, Thomas A., 2008. "Prediction market accuracy in the long run," International Journal of Forecasting, Elsevier, vol. 24(2), pages 285-300.
    22. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research:'s Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    23. Lian Jian & Rahul Sami, 2012. "Aggregation and Manipulation in Prediction Markets: Effects of Trading Mechanism and Information Distribution," Management Science, INFORMS, vol. 58(1), pages 123-140, January.
    Full references (including those not matched with items on IDEAS)

    More about this item


    Experiments; Mechanical turk; Electronic markets; Information aggregation;

    JEL classification:

    • C9 - Mathematical and Quantitative Methods - - Design of Experiments
    • D8 - Microeconomics - - Information, Knowledge, and Uncertainty


    Access and download statistics


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:grdene:v:27:y:2018:i:2:d:10.1007_s10726-018-9565-y. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Sonal Shukla) or (Springer Nature Abstracting and Indexing). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.