IDEAS home Printed from https://ideas.repec.org/a/eee/beexfi/v23y2019icp138-160.html

Web-based experimental economics software: How do they compare to desirable features?

Author

Listed:
  • Chan, Shu Wing
  • Schilizzi, Steven
  • Iftekhar, Md Sayed
  • Da Silva Rosa, Raymond

Abstract

Web-based experiments that cut across the lab vs. field distinction are increasingly popular with economists. However, non-standardized software features and services hinder comparability and replication. This study reviews a wide selection of experimental economics software packages and evaluates them against criteria based on the logistics and operational requirements of economic experiments. We find that oTree and SoPHIE rank highest across criteria, but Veconlab and classEx might be suitable for those with a dominant need for a large library of ready-made experiments. We find a portability gap: no presently available software allows portability of experiments across platforms because of technical complexity and the challenging coordination needs of experimental economists. As a result, experiments may be replicated only on the same platform or with the same software, but general replicability is slow and costly. This constrains the development of experimental economics as a replicable science.

Suggested Citation

  • Chan, Shu Wing & Schilizzi, Steven & Iftekhar, Md Sayed & Da Silva Rosa, Raymond, 2019. "Web-based experimental economics software: How do they compare to desirable features?," Journal of Behavioral and Experimental Finance, Elsevier, vol. 23(C), pages 138-160.
    • Handle: RePEc:eee:beexfi:v:23:y:2019:i:c:p:138-160
    DOI: 10.1016/j.jbef.2019.04.007
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S221463501830090X
    Download Restriction: no
    File URL: https://libkey.io/10.1016/j.jbef.2019.04.007?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    2. Paolacci, Gabriele & Chandler, Jesse & Ipeirotis, Panagiotis G., 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Cambridge University Press, vol. 5(5), pages 411-419, August.
    3. Ben Greiner, 2015. "Subject pool recruitment procedures: organizing experiments with ORSEE," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 1(1), pages 114-125, July.
    4. Paul Gertler & Sebastian Galiani & Mauricio Romero, 2018. "How to make replication the norm," Nature, Nature, vol. 554(7693), pages 417-419, February.
    5. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    6. Bock, Olaf & Baetge, Ingmar & Nicklisch, Andreas, 2014. "hroot: Hamburg Registration and Organization Online Tool," European Economic Review, Elsevier, vol. 71(C), pages 117-120.
    7. Giamattei, Marcus & Lambsdorff, Johann Graf, 2019. "classEx — an online tool for lab-in-the-field experiments with smartphones," Journal of Behavioral and Experimental Finance, Elsevier, vol. 22(C), pages 223-231.
    8. Seithe, Mirko, 2012. "Introducing the Bonn Experiment System (BoXS)," Bonn Econ Discussion Papers 01/2012, University of Bonn, Bonn Graduate School of Economics (BGSE).
    9. Chen, Daniel L. & Schonger, Martin & Wickens, Chris, 2016. "oTree—An open-source platform for laboratory, online, and field experiments," Journal of Behavioral and Experimental Finance, Elsevier, vol. 9(C), pages 88-97.
    10. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    11. Urs Fischbacher, 2007. "z-Tree: Zurich toolbox for ready-made economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 10(2), pages 171-178, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kumar, Satish & Rao, Sandeep & Goyal, Kirti & Goyal, Nisha, 2022. "Journal of Behavioral and Experimental Finance: A bibliometric overview," Journal of Behavioral and Experimental Finance, Elsevier, vol. 34(C).
    2. Marcus Giamattei & Kyanoush Seyed Yahosseini & Simon Gächter & Lucas Molleman, 2020. "LIONESS Lab: a free web-based platform for conducting interactive experiments online," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(1), pages 95-111, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Marcus Giamattei & Kyanoush Seyed Yahosseini & Simon Gächter & Lucas Molleman, 2020. "LIONESS Lab: a free web-based platform for conducting interactive experiments online," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(1), pages 95-111, June.
    2. Palan, Stefan & Schitter, Christian, 2018. "Prolific.ac—A subject pool for online experiments," Journal of Behavioral and Experimental Finance, Elsevier, vol. 17(C), pages 22-27.
    3. Molnar, Andras, 2019. "SMARTRIQS: A Simple Method Allowing Real-Time Respondent Interaction in Qualtrics Surveys," Journal of Behavioral and Experimental Finance, Elsevier, vol. 22(C), pages 161-169.
    4. Schwaiger, Rene & Hueber, Laura, 2021. "Do MTurkers exhibit myopic loss aversion?," Economics Letters, Elsevier, vol. 209(C).
    5. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    6. Mourelatos, Evangelos, 2023. "Does Mood affect Sexual and Gender Discrimination in Hiring Choices? Evidence from Online Experiments," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 106(C).
    7. Hyndman, Kyle & Walker, Matthew J., 2022. "Fairness and risk in ultimatum bargaining," Games and Economic Behavior, Elsevier, vol. 132(C), pages 90-105.
    8. Fumagalli, Elena & Rezaei, Sarah & Salomons, Anna, 2022. "OK computer: Worker perceptions of algorithmic recruitment," Research Policy, Elsevier, vol. 51(2).
    9. Mourelatos, Evangelos, 2021. "Personality and Ethics on Online Labor Markets: How mood influences ethical perceptions," EconStor Preprints 244735, ZBW - Leibniz Information Centre for Economics.
    10. Gächter, Simon & Kölle, Felix & Quercia, Simone, 2022. "Preferences and perceptions in Provision and Maintenance public goods," Games and Economic Behavior, Elsevier, vol. 135(C), pages 338-355.
    11. Kusterer, David & Sliwka, Dirk, 2022. "Social Preferences and Rating Biases in Subjective Performance Evaluations," IZA Discussion Papers 15496, IZA Network @ LISER.
    12. Florian Baumann & Tim Friehe & Pascal Langenbach, 2020. "Fines versus Damages: Experimental Evidence on Care Investments," Discussion Paper Series of the Max Planck Institute for Behavioral Economics 2020_08, Max Planck Institute for Behavioral Economics, revised Mar 2024.
    13. Roggenkamp, Hauke, 2025. "A comment on ‘growth and inequality in public good provision’: Testing the robustness and generalizability of dynamic public good games," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 115(C).
    14. Rutledge, Robert M. & Alladi, Vinayak & Cheung, Stephen L., 2025. "Price expectations and reference-dependent preferences," Journal of Economic Behavior & Organization, Elsevier, vol. 231(C).
    15. Haas, Nicholas & Hassan, Mazen & Mansour, Sarah & Morton, Rebecca B., 2021. "Polarizing information and support for reform," Journal of Economic Behavior & Organization, Elsevier, vol. 185(C), pages 883-901.
    16. Hindsley, Paul & McEvoy, David M. & Morgan, O. Ashton, 2020. "Consumer Demand for Ethical Products and the Role of Cultural Worldviews: The Case of Direct-Trade Coffee," Ecological Economics, Elsevier, vol. 177(C).
    17. Boukouras, Aristotelis & Jennings, Will & Li, Lunzheng & Maniadis, Zacharias, 2023. "Can biased polls distort electoral results? Evidence from the lab," European Journal of Political Economy, Elsevier, vol. 78(C).
    18. Dato, Simon & Feess, Eberhard & Nieken, Petra, 2019. "Lying and reciprocity," Games and Economic Behavior, Elsevier, vol. 118(C), pages 193-218.
    19. Wladislaw Mill & Cornelius Schneider, 2023. "The Bright Side of Tax Evasion," CESifo Working Paper Series 10615, CESifo.
    20. Danková, Katarína & Servátka, Maroš, 2019. "Gender robustness of overconfidence and excess entry," Journal of Economic Psychology, Elsevier, vol. 72(C), pages 179-199.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    JEL classification:

    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • C88 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Other Computer Software

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:beexfi:v:23:y:2019:i:c:p:138-160. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/journal-of-behavioral-and-experimental-finance .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.