IDEAS home Printed from https://ideas.repec.org/p/gre/wpaper/2020-28.html
   My bibliography  Save this paper

The Use of Experimental Methods by IS Scholars: An Illustrated Typology

Author

Listed:
  • Marta Ballatore

    (GREDEG CNRS
    Université Côte d'Azur, France)

  • Lise Arena

    (GREDEG CNRS
    Université Côte d'Azur, France)

  • Agnès Festré

    (GREDEG CNRS
    Université Côte d'Azur, France)

Abstract

This article aims at making an updated typology of recent experimental studies in the IS literature on the period 1999-2019. Based on a full-text search within the Association for Information Systems (AIS) “basket” of eight top IS journals (EJIS, ISR, JAIS, ISJ, JIT, JMIS, JSIS and MISQ), this research gathered 392 articles and highlights the use of 5 different types of experiments in IS, mainly: artificial simulations, laboratory experiments, field experiments, online experiments (scenario simulation game-based; brainstorming-based. . . ) and natural experiments. Each category is discussed through the perspective of its degree of control, and technological realism. Results show the significant predominance of laboratory experiments over field and natural experiments on the period. This, in turn, stresses the preferred tendency followed by IS scholars to perceive experimental methods as a way to control the source of variations of variables under study. In addition, this paper provides a better understanding of the context of use of a specific experimental method. Overall, it is shown that laboratory experiments (including scenario-based lab experiments) are mainly used, in a deterministic manner, to assess or test the impact of an IS on human decision-making or behaviour. By contrast, artificial simulations experiments are more appropriate to study emergent phenomena and to make predictions, often providing key insights about quality and effectiveness of IS.

Suggested Citation

  • Marta Ballatore & Lise Arena & Agnès Festré, 2020. "The Use of Experimental Methods by IS Scholars: An Illustrated Typology," GREDEG Working Papers 2020-28, Groupe de REcherche en Droit, Economie, Gestion (GREDEG CNRS), Université Côte d'Azur, France.
  • Handle: RePEc:gre:wpaper:2020-28
    as

    Download full text from publisher

    File URL: http://195.220.190.85/GREDEG-WP-2020-28.pdf
    File Function: First version, 2020
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Kar Yan Tam & Shuk Ying Ho, 2005. "Web Personalization as a Persuasion Strategy: An Elaboration Likelihood Model Perspective," Information Systems Research, INFORMS, vol. 16(3), pages 271-291, September.
    2. Ray M. Chang & Wonseok Oh & Alain Pinsonneault & Dowan Kwon, 2010. "A Network Perspective of Digital Competition in Online Advertising Industries: A Simulation-Based Approach," Information Systems Research, INFORMS, vol. 21(3), pages 571-593, September.
    3. Anuj Kumar & Kartik Hosanagar, 2019. "Measuring the Value of Recommendation Links on Product Demand," Information Systems Research, INFORMS, vol. 30(3), pages 819-838, September.
    4. Amy Wenxuan Ding & Shibo Li & Patrali Chatterjee, 2015. "Learning User Real-Time Intent for Optimal Dynamic Web Page Transformation," Information Systems Research, INFORMS, vol. 26(2), pages 339-359, June.
    5. Angelika Dimoka & Paul A. Pavlou & Fred D. Davis, 2011. "Research Commentary ---NeuroIS: The Potential of Cognitive Neuroscience for Information Systems Research," Information Systems Research, INFORMS, vol. 22(4), pages 687-702, December.
    6. Weiquan Wang & May Wang, 2019. "Effects of Sponsorship Disclosure on Perceived Integrity of Biased Recommendation Agents: Psychological Contract Violation and Knowledge-Based Trust Perspectives," Information Systems Research, INFORMS, vol. 30(2), pages 507-522, June.
    7. Matthew J. Hashim & Karthik N. Kannan & Sandra Maximiano, 2017. "Information Feedback, Targeting, and Coordination: An Experimental Study," Information Systems Research, INFORMS, vol. 28(2), pages 289-308, June.
    8. Vernon L. Smith, 1994. "Economics in the Laboratory," Journal of Economic Perspectives, American Economic Association, vol. 8(1), pages 113-131, Winter.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bo Zhou & Tianxin Zou, 2023. "Competing for Recommendations: The Strategic Impact of Personalized Product Recommendations in Online Marketplaces," Marketing Science, INFORMS, vol. 42(2), pages 360-376, March.
    2. Xueming Luo & Xianghua Lu & Jing Li, 2019. "When and How to Leverage E-commerce Cart Targeting: The Relative and Moderated Effects of Scarcity and Price Incentives with a Two-Stage Field Experiment and Causal Forest Optimization," Information Systems Research, INFORMS, vol. 30(4), pages 1203-1227, December.
    3. Gabriele Camera & Cary Deck & David Porter, 2020. "Do economic inequalities affect long-run cooperation and prosperity?," Experimental Economics, Springer;Economic Science Association, vol. 23(1), pages 53-83, March.
    4. Mahesh Balan U & Saji K. Mathew, 2021. "Personalize, Summarize or Let them Read? A Study on Online Word of Mouth Strategies and Consumer Decision Process," Information Systems Frontiers, Springer, vol. 23(3), pages 627-647, June.
    5. Nitin Walia & Mark Srite & Wendy Huddleston, 2016. "Eyeing the web interface: the influence of price, product, and personal involvement," Electronic Commerce Research, Springer, vol. 16(3), pages 297-333, September.
    6. Deck, Cary & Murphy, James J., 2019. "Donors change both their level and pattern of giving in response to contests among charities," European Economic Review, Elsevier, vol. 112(C), pages 91-106.
    7. Khim-Yong Goh & Cheng-Suang Heng & Zhijie Lin, 2013. "Social Media Brand Community and Consumer Behavior: Quantifying the Relative Impact of User- and Marketer-Generated Content," Information Systems Research, INFORMS, vol. 24(1), pages 88-107, March.
    8. Cary Deck & Erik O. Kimbrough, 2017. "Experimenting with Contests for Experimentation," Southern Economic Journal, John Wiley & Sons, vol. 84(2), pages 391-406, October.
    9. Cary Deck & Maroš Servátka & Steven Tucker, 2013. "An examination of the effect of messages on cooperation under double-blind and single-blind payoff procedures," Experimental Economics, Springer;Economic Science Association, vol. 16(4), pages 597-607, December.
    10. Ackert, Lucy F. & Church, Bryan K. & Zhang, Ping, 2004. "Asset prices and informed traders' abilities: Evidence from experimental asset markets," Accounting, Organizations and Society, Elsevier, vol. 29(7), pages 609-626, October.
    11. Zizzo, Daniel John, 2013. "Claims and confounds in economic experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 93(C), pages 186-195.
    12. Nasim Mousavi & Panagiotis Adamopoulos & Jesse Bockstedt, 2023. "The Decoy Effect and Recommendation Systems," Information Systems Research, INFORMS, vol. 34(4), pages 1533-1553, December.
    13. Nicolas Jacquemet & Alexander James & Stéphane Luchini & Jason F. Shogren, 2017. "Referenda Under Oath," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 67(3), pages 479-504, July.
    14. Tibert Verhagen & Daniel Bloemers, 2018. "Exploring the cognitive and affective bases of online purchase intentions: a hierarchical test across product types," Electronic Commerce Research, Springer, vol. 18(3), pages 537-561, September.
    15. Roy Chen & Yan Chen & Yohanes E. Riyanto, 2021. "Best practices in replication: a case study of common information in coordination games," Experimental Economics, Springer;Economic Science Association, vol. 24(1), pages 2-30, March.
    16. Kim, Jae Kyeong & Kim, Hyea Kyeong & Oh, Hee Young & Ryu, Young U., 2010. "A group recommendation system for online communities," International Journal of Information Management, Elsevier, vol. 30(3), pages 212-219.
    17. Dwenger, Nadja & Lohse, Tim, 2019. "Do individuals successfully cover up their lies? Evidence from a compliance experiment," Journal of Economic Psychology, Elsevier, vol. 71(C), pages 74-87.
    18. Hermann Garbers, "undated". "Agents' Rationality and the CHF/USD Exchange Rate, Part II," IEW - Working Papers 169, Institute for Empirical Research in Economics - University of Zurich.
    19. Tian, Ye & Li, Yudi & Sun, Jian, 2022. "Stick or carrot for traffic demand management? Evidence from experimental economics," Transportation Research Part A: Policy and Practice, Elsevier, vol. 160(C), pages 235-254.
    20. Morten Søberg, 2002. "The Duhem-Quine thesis and experimental economics. A reinterpretation," Discussion Papers 329, Statistics Norway, Research Department.

    More about this item

    Keywords

    Methodology; IS; Meta-research; Experimental method; Behavioural Science; Experimental Economics;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gre:wpaper:2020-28. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Patrice Bougette (email available below). General contact details of provider: https://edirc.repec.org/data/credcfr.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.