IDEAS home Printed from https://ideas.repec.org/a/spr/grdene/v27y2018i2d10.1007_s10726-018-9565-y.html
   My bibliography  Save this article

Crowd Labor Markets as Platform for Group Decision and Negotiation Research: A Comparison to Laboratory Experiments

Author

Listed:
  • Florian Teschner

    (Karlsruhe Institute of Technology (KIT))

  • Henner Gimpel

    (University of Augsburg)

Abstract

Crowd labor markets such as Amazon Mechanical Turk (MTurk) have emerged as popular platforms where researchers can relatively inexpensively and easily run web-based experiments. Some work even suggests that MTurk can be used to run large-scale field experiments in which groups of participants interact synchronously in real-time such as electronic markets. Besides technical issues, several methodological questions arise and lead to the question of how results from MTurk and laboratory experiments compare. Our data shows comparable results between MTurk and a standard lab setting with student subjects in a controlled environment when running rather simple individual decision tasks. However, our data shows stark differences in results between the experimental settings for a rather complex market experiment. Each experimental setting—lab and MTurk—has its own benefits and drawbacks; which of the two settings is better suited for a specific experiment depends on the theory or artifact to be tested. We discuss potential causes for differences (language understanding, education, cognition and context) that we cannot control for and provide guidance for the selection of the appropriate setting for an experiment. In any case, researchers studying complex artifacts like group decisions or markets should not prematurely adopt MTurk based on extant literature regarding comparable results across experimental settings for rather simple tasks.

Suggested Citation

  • Florian Teschner & Henner Gimpel, 2018. "Crowd Labor Markets as Platform for Group Decision and Negotiation Research: A Comparison to Laboratory Experiments," Group Decision and Negotiation, Springer, vol. 27(2), pages 197-214, April.
    • Handle: RePEc:spr:grdene:v:27:y:2018:i:2:d:10.1007_s10726-018-9565-y
    DOI: 10.1007/s10726-018-9565-y
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10726-018-9565-y
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10726-018-9565-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    2. Martin Spann & Bernd Skiera, 2003. "Internet-Based Virtual Stock Markets for Business Forecasting," Management Science, INFORMS, vol. 49(10), pages 1310-1326, October.
    3. Chandler, Dana & Kapelner, Adam, 2013. "Breaking monotony with meaning: Motivation in crowdsourcing markets," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 123-133.
    4. Antonio Ferreira & Pedro Antunes & Valeria Herskovic, 2011. "Improving Group Attention: An Experiment with Synchronous Brainstorming," Group Decision and Negotiation, Springer, vol. 20(5), pages 643-666, September.
    5. Justin Wolfers & Eric Zitzewitz, 2004. "Prediction Markets," Journal of Economic Perspectives, American Economic Association, vol. 18(2), pages 107-126, Spring.
    6. Joyce E. Berg & Thomas A. Rietz, 2003. "Prediction Markets as Decision Support Systems," Information Systems Frontiers, Springer, vol. 5(1), pages 79-93, January.
    7. Gregory Kersten & Sunil Noronha, 1999. "Negotiation via the World Wide Web: A Cross-cultural Study of Decision Making," Group Decision and Negotiation, Springer, vol. 8(3), pages 251-279, May.
    8. Bennouri, Moez & Gimpel, Henner & Robert, Jacques, 2011. "Measuring the impact of information aggregation mechanisms: An experimental investigation," Journal of Economic Behavior & Organization, Elsevier, vol. 78(3), pages 302-318, May.
    9. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    10. Fair, Ray C & Shiller, Robert J, 1989. "The Informational Context of Ex Ante Forecasts," The Review of Economics and Statistics, MIT Press, vol. 71(2), pages 325-331, May.
    11. Paul J. Healy & Sera Linardi & J. Richard Lowery & John O. Ledyard, 2010. "Prediction Markets: Alternative Mechanisms for Complex Environments with Few Traders," Management Science, INFORMS, vol. 56(11), pages 1977-1996, November.
    12. Ofra Amir & David G Rand & Ya'akov Kobi Gal, 2012. "Economic Games on the Internet: The Effect of $1 Stakes," PLOS ONE, Public Library of Science, vol. 7(2), pages 1-4, February.
    13. Chen, Daniel L. & Horton, John J., 2016. "Are Online Labor Markets Spot Markets for Tasks?: A Field Experiment on the Behavioral Response to Wage Cuts," IAST Working Papers 16-37, Institute for Advanced Study in Toulouse (IAST).
    14. Ledyard, John & Hanson, Robin & Ishikida, Takashi, 2009. "An experimental test of combinatorial information markets," Journal of Economic Behavior & Organization, Elsevier, vol. 69(2), pages 182-189, February.
    15. Florian Teschner & David Rothschild & Henner Gimpel, 2017. "Manipulation in Conditional Decision Markets," Group Decision and Negotiation, Springer, vol. 26(5), pages 953-971, September.
    16. Logan S. Casey & Jesse Chandler & Adam Seth Levine & Andrew Proctor & Dara Z. Strolovitch, 2017. "Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection," SAGE Open, , vol. 7(2), pages 21582440177, June.
    17. Mullinix, Kevin J. & Leeper, Thomas J. & Druckman, James N. & Freese, Jeremy, 2015. "The Generalizability of Survey Experiments," Journal of Experimental Political Science, Cambridge University Press, vol. 2(2), pages 109-138, January.
    18. Plott, Charles R & Sunder, Shyam, 1988. "Rational Expectations and the Aggregation of Diverse Information in Laboratory Security Markets," Econometrica, Econometric Society, vol. 56(5), pages 1085-1118, September.
    19. T. S. Breusch & A. R. Pagan, 1980. "The Lagrange Multiplier Test and its Applications to Model Specification in Econometrics," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 47(1), pages 239-253.
    20. Alain Pinsonneault & Henri Barki & R. Brent Gallupe & Norberto Hoppen, 1999. "Electronic Brainstorming: The Illusion of Productivity," Information Systems Research, INFORMS, vol. 10(2), pages 110-133, June.
    21. Robin Hanson, 2003. "Combinatorial Information Market Design," Information Systems Frontiers, Springer, vol. 5(1), pages 107-119, January.
    22. Brad M. Barber & Terrance Odean, 2000. "Trading Is Hazardous to Your Wealth: The Common Stock Investment Performance of Individual Investors," Journal of Finance, American Finance Association, vol. 55(2), pages 773-806, April.
    23. Shane Frederick, 2005. "Cognitive Reflection and Decision Making," Journal of Economic Perspectives, American Economic Association, vol. 19(4), pages 25-42, Fall.
    24. Jim Lavoie, 2009. "The Innovation Engine at Rite-Solutions: Lessons from the CEO," Journal of Prediction Markets, University of Buckingham Press, vol. 3(1), pages 1-11, April.
    25. Berg, Joyce E. & Nelson, Forrest D. & Rietz, Thomas A., 2008. "Prediction market accuracy in the long run," International Journal of Forecasting, Elsevier, vol. 24(2), pages 285-300.
    26. Tim Straub & Henner Gimpel & Florian Teschner & Christof Weinhardt, 2015. "How (not) to Incent Crowd Workers," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 57(3), pages 167-179, June.
    27. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    28. Lian Jian & Rahul Sami, 2012. "Aggregation and Manipulation in Prediction Markets: Effects of Trading Mechanism and Information Distribution," Management Science, INFORMS, vol. 58(1), pages 123-140, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Gert Jan Hofstede & Catholijn M. Jonker & Tim Verwaart & Neil Yorke-Smith, 2019. "The Lemon Car Game Across Cultures: Evidence of Relational Rationality," Group Decision and Negotiation, Springer, vol. 28(5), pages 849-877, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Florian Teschner & David Rothschild & Henner Gimpel, 2017. "Manipulation in Conditional Decision Markets," Group Decision and Negotiation, Springer, vol. 26(5), pages 953-971, September.
    2. Lian Jian & Rahul Sami, 2012. "Aggregation and Manipulation in Prediction Markets: Effects of Trading Mechanism and Information Distribution," Management Science, INFORMS, vol. 58(1), pages 123-140, January.
    3. Galanis, S. & Ioannou, C. & Kotronis, S., 2019. "Information Aggregation Under Ambiguity: Theory and Experimental Evidence," Working Papers 20/05, Department of Economics, City University London.
    4. Ledyard, John & Hanson, Robin & Ishikida, Takashi, 2009. "An experimental test of combinatorial information markets," Journal of Economic Behavior & Organization, Elsevier, vol. 69(2), pages 182-189, February.
    5. Snowberg, Erik & Wolfers, Justin & Zitzewitz, Eric, 2013. "Prediction Markets for Economic Forecasting," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 657-687, Elsevier.
    6. Page, Lionel & Siemroth, Christoph, 2017. "An experimental analysis of information acquisition in prediction markets," Games and Economic Behavior, Elsevier, vol. 101(C), pages 354-378.
    7. Edoardo Gaffeo, 2013. "Using information markets in grantmaking. An assessment of the issues involved and an application to Italian banking foundations," DEM Discussion Papers 2013/08, Department of Economics and Management.
    8. Justin Wolfers & Eric Zitzewitz, 2004. "Prediction Markets," Journal of Economic Perspectives, American Economic Association, vol. 18(2), pages 107-126, Spring.
    9. Forsell, Eskil & Viganola, Domenico & Pfeiffer, Thomas & Almenberg, Johan & Wilson, Brad & Chen, Yiling & Nosek, Brian A. & Johannesson, Magnus & Dreber, Anna, 2019. "Predicting replication outcomes in the Many Labs 2 study," Journal of Economic Psychology, Elsevier, vol. 75(PA).
    10. Karimi, Majid & Zaerpour, Nima, 2022. "Put your money where your forecast is: Supply chain collaborative forecasting with cost-function-based prediction markets," European Journal of Operational Research, Elsevier, vol. 300(3), pages 1035-1049.
    11. Patrick Buckley & Fergal O’Brien, 0. "The effect of malicious manipulations on prediction market accuracy," Information Systems Frontiers, Springer, vol. 0, pages 1-13.
    12. Prissé, Benjamin & Jorrat, Diego, 2022. "Lab vs online experiments: No differences," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    13. Paul J. Healy & Sera Linardi & J. Richard Lowery & John O. Ledyard, 2010. "Prediction Markets: Alternative Mechanisms for Complex Environments with Few Traders," Management Science, INFORMS, vol. 56(11), pages 1977-1996, November.
    14. Bennouri, Moez & Gimpel, Henner & Robert, Jacques, 2011. "Measuring the impact of information aggregation mechanisms: An experimental investigation," Journal of Economic Behavior & Organization, Elsevier, vol. 78(3), pages 302-318, May.
    15. Ho Cheung Brian Lee & Jan Stallaert & Ming Fan, 2020. "Anomalies in Probability Estimates for Event Forecasting on Prediction Markets," Production and Operations Management, Production and Operations Management Society, vol. 29(9), pages 2077-2095, September.
    16. Patrick Buckley & Fergal O’Brien, 2017. "The effect of malicious manipulations on prediction market accuracy," Information Systems Frontiers, Springer, vol. 19(3), pages 611-623, June.
    17. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    18. Johannes G. Jaspersen & Marc A. Ragin & Justin R. Sydnor, 2022. "Insurance demand experiments: Comparing crowdworking to the lab," Journal of Risk & Insurance, The American Risk and Insurance Association, vol. 89(4), pages 1077-1107, December.
    19. Lawrence Choo & Todd R. Kaplan & Ro’i Zultan, 2022. "Manipulation and (Mis)trust in Prediction Markets," Management Science, INFORMS, vol. 68(9), pages 6716-6732, September.
    20. Abraham Othman & Tuomas Sandholm, 2013. "The Gates Hillman prediction market," Review of Economic Design, Springer;Society for Economic Design, vol. 17(2), pages 95-128, June.

    More about this item

    Keywords

    Experiments; Mechanical turk; Electronic markets; Information aggregation;
    All these keywords.

    JEL classification:

    • C9 - Mathematical and Quantitative Methods - - Design of Experiments
    • D8 - Microeconomics - - Information, Knowledge, and Uncertainty

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:grdene:v:27:y:2018:i:2:d:10.1007_s10726-018-9565-y. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.