IDEAS home Printed from https://ideas.repec.org/a/eee/beexfi/v23y2019icp138-160.html

Web-based experimental economics software: How do they compare to desirable features?

Author

Listed:
  • Chan, Shu Wing
  • Schilizzi, Steven
  • Iftekhar, Md Sayed
  • Da Silva Rosa, Raymond

Abstract

Web-based experiments that cut across the lab vs. field distinction are increasingly popular with economists. However, non-standardized software features and services hinder comparability and replication. This study reviews a wide selection of experimental economics software packages and evaluates them against criteria based on the logistics and operational requirements of economic experiments. We find that oTree and SoPHIE rank highest across criteria, but Veconlab and classEx might be suitable for those with a dominant need for a large library of ready-made experiments. We find a portability gap: no presently available software allows portability of experiments across platforms because of technical complexity and the challenging coordination needs of experimental economists. As a result, experiments may be replicated only on the same platform or with the same software, but general replicability is slow and costly. This constrains the development of experimental economics as a replicable science.

Suggested Citation

  • Chan, Shu Wing & Schilizzi, Steven & Iftekhar, Md Sayed & Da Silva Rosa, Raymond, 2019. "Web-based experimental economics software: How do they compare to desirable features?," Journal of Behavioral and Experimental Finance, Elsevier, vol. 23(C), pages 138-160.
    • Handle: RePEc:eee:beexfi:v:23:y:2019:i:c:p:138-160
    DOI: 10.1016/j.jbef.2019.04.007
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S221463501830090X
    Download Restriction: no
    File URL: https://libkey.io/10.1016/j.jbef.2019.04.007?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    2. Urs Fischbacher, 2007. "z-Tree: Zurich toolbox for ready-made economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 10(2), pages 171-178, June.
    3. Chen, Daniel L. & Schonger, Martin & Wickens, Chris, 2016. "oTree—An open-source platform for laboratory, online, and field experiments," Journal of Behavioral and Experimental Finance, Elsevier, vol. 9(C), pages 88-97.
    4. Paul Gertler & Sebastian Galiani & Mauricio Romero, 2018. "How to make replication the norm," Nature, Nature, vol. 554(7693), pages 417-419, February.
    5. Paolacci, Gabriele & Chandler, Jesse & Ipeirotis, Panagiotis G., 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Cambridge University Press, vol. 5(5), pages 411-419, August.
    6. Seithe, Mirko, 2012. "Introducing the Bonn Experiment System (BoXS)," Bonn Econ Discussion Papers 01/2012, University of Bonn, Bonn Graduate School of Economics (BGSE).
    7. Ben Greiner, 2015. "Subject pool recruitment procedures: organizing experiments with ORSEE," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 1(1), pages 114-125, July.
    8. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    9. Giamattei, Marcus & Lambsdorff, Johann Graf, 2019. "classEx — an online tool for lab-in-the-field experiments with smartphones," Journal of Behavioral and Experimental Finance, Elsevier, vol. 22(C), pages 223-231.
    10. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    11. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    12. Bock, Olaf & Baetge, Ingmar & Nicklisch, Andreas, 2014. "hroot: Hamburg Registration and Organization Online Tool," European Economic Review, Elsevier, vol. 71(C), pages 117-120.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kumar, Satish & Rao, Sandeep & Goyal, Kirti & Goyal, Nisha, 2022. "Journal of Behavioral and Experimental Finance: A bibliometric overview," Journal of Behavioral and Experimental Finance, Elsevier, vol. 34(C).
    2. Marcus Giamattei & Kyanoush Seyed Yahosseini & Simon Gächter & Lucas Molleman, 2020. "LIONESS Lab: a free web-based platform for conducting interactive experiments online," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(1), pages 95-111, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Marcus Giamattei & Kyanoush Seyed Yahosseini & Simon Gächter & Lucas Molleman, 2020. "LIONESS Lab: a free web-based platform for conducting interactive experiments online," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(1), pages 95-111, June.
    2. Molnar, Andras, 2019. "SMARTRIQS: A Simple Method Allowing Real-Time Respondent Interaction in Qualtrics Surveys," Journal of Behavioral and Experimental Finance, Elsevier, vol. 22(C), pages 161-169.
    3. Gächter, Simon & Kölle, Felix & Quercia, Simone, 2022. "Preferences and perceptions in Provision and Maintenance public goods," Games and Economic Behavior, Elsevier, vol. 135(C), pages 338-355.
    4. Roggenkamp, Hauke, 2025. "A comment on ‘growth and inequality in public good provision’: Testing the robustness and generalizability of dynamic public good games," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 115(C).
    5. Palan, Stefan & Schitter, Christian, 2018. "Prolific.ac—A subject pool for online experiments," Journal of Behavioral and Experimental Finance, Elsevier, vol. 17(C), pages 22-27.
    6. Florian Baumann & Tim Friehe & Pascal Langenbach, 2020. "Fines versus Damages: Experimental Evidence on Care Investments," Discussion Paper Series of the Max Planck Institute for Research on Collective Goods 2020_08, Max Planck Institute for Research on Collective Goods, revised Mar 2024.
    7. Rutledge, Robert M. & Alladi, Vinayak & Cheung, Stephen L., 2025. "Price expectations and reference-dependent preferences," Journal of Economic Behavior & Organization, Elsevier, vol. 231(C).
    8. Lorko, Matej & Servátka, Maroš & Zhang, Le, 2023. "Hidden inefficiency: Strategic inflation of project schedules," Journal of Economic Behavior & Organization, Elsevier, vol. 206(C), pages 313-326.
    9. Boukouras, Aristotelis & Jennings, Will & Li, Lunzheng & Maniadis, Zacharias, 2023. "Can biased polls distort electoral results? Evidence from the lab," European Journal of Political Economy, Elsevier, vol. 78(C).
    10. König-Kersting, Christian & Trautmann, Stefan T. & Vlahu, Razvan, 2022. "Bank instability: Interbank linkages and the role of disclosure," Journal of Banking & Finance, Elsevier, vol. 134(C).
    11. Fischbacher, Urs & Hausfeld, Jan & Renerte, Baiba, 2022. "Strategic incentives undermine gaze as a signal of prosocial motives," Games and Economic Behavior, Elsevier, vol. 136(C), pages 63-91.
    12. Schmidt, Robert J., 2019. "Do injunctive or descriptive social norms elicited using coordination games better explain social preferences?," Working Papers 0668, University of Heidelberg, Department of Economics.
    13. Fehr, Dietmar & Rau, Hannes & Trautmann, Stefan T. & Xu, Yilong, 2020. "Inequality, fairness and social capital," European Economic Review, Elsevier, vol. 129(C).
    14. Fellner-Röhling, Gerlinde & Hromek, Kristijan & Kleinknecht, Janina & Ludwig, Sandra, 2023. "Reciprocal reactions to (in)transparent task assignments: An experimental investigation," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 106(C).
    15. Florian Engl & Arno Riedl & Roberto Weber, 2021. "Spillover Effects of Institutions on Cooperative Behavior, Preferences, and Beliefs," American Economic Journal: Microeconomics, American Economic Association, vol. 13(4), pages 261-299, November.
    16. Romaniuc, Rustam & Guido, Andrea & Baudry, Pierre & Bazart, Cécile & Berger, Loïc & Berlin, Noémi & Bonein, Aurélie & Bouhlel, Imen & My, Kene Boun & Chessa, Michela & Crosetto, Paolo & Dagorn, Etienn, 2025. "The limits of behavioral nudges to increase youth turnout: Experimental evidence from two French elections," Journal of Economic Behavior & Organization, Elsevier, vol. 236(C).
    17. Heike Hennig‐Schmidt & Hendrik Jürges & Daniel Wiesen, 2019. "Dishonesty in health care practice: A behavioral experiment on upcoding in neonatology," Health Economics, John Wiley & Sons, Ltd., vol. 28(3), pages 319-338, March.
    18. Schwaiger, Rene & Hueber, Laura, 2021. "Do MTurkers exhibit myopic loss aversion?," Economics Letters, Elsevier, vol. 209(C).
    19. Despoina Alempaki & Emina Canic & Timothy L. Mullett & William J. Skylark & Chris Starmer & Neil Stewart & Fabio Tufano, 2019. "Reexamining How Utility and Weighting Functions Get Their Shapes: A Quasi-Adversarial Collaboration Providing a New Interpretation," Management Science, INFORMS, vol. 65(10), pages 4841-4862, October.
    20. Nisvan Erkal & Lata Gangadharan & Boon Han Koh, 2022. "By chance or by choice? Biased attribution of others’ outcomes when social preferences matter," Experimental Economics, Springer;Economic Science Association, vol. 25(2), pages 413-443, April.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    JEL classification:

    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • C88 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Other Computer Software

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:beexfi:v:23:y:2019:i:c:p:138-160. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/journal-of-behavioral-and-experimental-finance .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.