IDEAS home Printed from https://ideas.repec.org/a/spr/elmark/v33y2023i1d10.1007_s12525-023-00662-3.html
   My bibliography  Save this article

Enhancing conversational agents for successful operation: A multi-perspective evaluation approach for continuous improvement

Author

Listed:
  • Tom Lewandowski

    (University of Hamburg)

  • Emir Kučević

    (University of Hamburg)

  • Stephan Leible

    (University of Hamburg)

  • Mathis Poser

    (University of Hamburg)

  • Tilo Böhmann

    (University of Hamburg)

Abstract

Contemporary organizations increasingly adopt conversational agents (CAs) as intelligent and natural language-based solutions for providing services and information. CAs offer new forms of personalization, speed, (cost-)effectiveness, and automation. However, despite their hype in research and practice, many organizations still fail to seize CAs’ potential because they lack knowledge of how to evaluate and improve the quality of CAs to sustain them in organizational operations. We aim to fill this knowledge gap by conducting a design science research project in which we aggregate insights from the literature and practice to derive an applicable set of quality criteria for CAs. Our article contributes to CA research and guides practitioners by providing a blueprint to structure the evaluation of CAs and to discover areas for systematic improvement.

Suggested Citation

  • Tom Lewandowski & Emir Kučević & Stephan Leible & Mathis Poser & Tilo Böhmann, 2023. "Enhancing conversational agents for successful operation: A multi-perspective evaluation approach for continuous improvement," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-20, December.
    • Handle: RePEc:spr:elmark:v:33:y:2023:i:1:d:10.1007_s12525-023-00662-3
    DOI: 10.1007/s12525-023-00662-3
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s12525-023-00662-3
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s12525-023-00662-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Stephan Diederich & Alfred Benedikt Brendel & Lutz M. Kolbe, 2020. "Designing Anthropomorphic Enterprise Conversational Agents," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 62(3), pages 193-209, June.
    2. John Venable & Jan Pries-Heje & Richard Baskerville, 2016. "FEDS: a Framework for Evaluation in Design Science Research," European Journal of Information Systems, Taylor & Francis Journals, vol. 25(1), pages 77-89, January.
    3. Handan ÇAM, 2017. "3rd International conference on social sciences & education research," Journal of Economic and Social Thought, KSP Journals, vol. 4(3), pages 352-353, Seprember.
    4. Severin Weiler & Christian Matt & Thomas Hess, 2022. "Immunizing with information – Inoculation messages against conversational agents’ response failures," Electronic Markets, Springer;IIM University of St. Gallen, vol. 32(1), pages 239-258, March.
    5. Mengjun Li & Ayoung Suh, 2022. "Anthropomorphism in AI-enabled technology: A literature review," Electronic Markets, Springer;IIM University of St. Gallen, vol. 32(4), pages 2245-2275, December.
    6. Feng, Shengjia & Buxmann, Peter, 2020. "My Virtual Colleague: A State-of-the-Art Analysis of Conversational Agents for the Workplace," Publications of Darmstadt Technical University, Institute for Business Studies (BWL) 118817, Darmstadt Technical University, Department of Business Administration, Economics and Law, Institute for Business Studies (BWL).
    7. Giulia Pavon & Lars Meyer-Waarden & Thanida Poocharoentou & Piyanut Prayatsup & Maelis Ratinaud & Sara Torne, 2020. "How Service Quality Influences Customer Acceptance and Usage of Chatbots?," Post-Print hal-03145866, HAL.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Darima Fotheringham & Michael A. Wiles, 2023. "The effect of implementing chatbot customer service on stock returns: an event study analysis," Journal of the Academy of Marketing Science, Springer, vol. 51(4), pages 802-822, July.
    2. Raffaele Fabio Ciriello & Alexandra Cecilie Gjøl Torbensen & Magnus Rotvit Perlt Hansen & Christoph Müller-Bloch, 2023. "Blockchain-based digital rights management systems: Design principles for the music industry," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-21, December.
    3. Tao Zhang & Chao Feng & Hui Chen & Junjie Xian, 2022. "Calming the customers by AI: Investigating the role of chatbot acting-cute strategies in soothing negative customer emotions," Electronic Markets, Springer;IIM University of St. Gallen, vol. 32(4), pages 2277-2292, December.
    4. Tim Kollmer & Andreas Eckhardt, 2023. "Dark Patterns," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 65(2), pages 201-208, April.
    5. Justina Sidlauskiene & Yannick Joye & Vilte Auruskeviciene, 2023. "AI-based chatbots in conversational commerce and their effects on product and price perceptions," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-21, December.
    6. Hendrik Haße & Hendrik Valk & Frederik Möller & Boris Otto, 2022. "Design Principles for Shared Digital Twins in Distributed Systems," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 64(6), pages 751-772, December.
    7. Ulrich Gnewuch & Stefan Morana & Marc T. P. Adam & Alexander Maedche, 2022. "Opposing Effects of Response Time in Human–Chatbot Interaction," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 64(6), pages 773-791, December.
    8. Joshua Holstein & Max Schemmer & Johannes Jakubik & Michael Vössing & Gerhard Satzger, 2023. "Sanitizing data for analysis: Designing systems for data understanding," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-18, December.
    9. Laleh Rafati & Geert Poels, 2017. "Value-Driven Strategic Sourcing Based on Service-Dominant Logic," Service Science, INFORMS, vol. 9(4), pages 275-287, December.
    10. Thuy Duong Oesterreich & Eduard Anton & Julian Schuir & Alexander Brehm & Frank Teuteberg, 2023. "How can I help you? Design principles for task-oriented speech dialog systems in customer service," Information Systems and e-Business Management, Springer, vol. 21(1), pages 37-79, March.
    11. Rainer Alt, 2022. "Electronic Markets on platform dualities," Electronic Markets, Springer;IIM University of St. Gallen, vol. 32(1), pages 1-10, March.
    12. Julius Peter Landwehr & Niklas Kühl & Jannis Walk & Mario Gnädig, 2022. "Design Knowledge for Deep-Learning-Enabled Image-Based Decision Support Systems," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 64(6), pages 707-728, December.
    13. Christian Meske & Enrico Bunde, 2023. "Design Principles for User Interfaces in AI-Based Decision Support Systems: The Case of Explainable Hate Speech Detection," Information Systems Frontiers, Springer, vol. 25(2), pages 743-773, April.
    14. Tobias Wulfert, 2023. "Boundary resource management in innovation ecosystems: The case of e-commerce," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-27, December.
    15. Zogaj, Adnan & Mähner, Philipp M. & Yang, Linyu & Tscheulin, Dieter K., 2023. "It’s a Match! The effects of chatbot anthropomorphization and chatbot gender on consumer behavior," Journal of Business Research, Elsevier, vol. 155(PA).
    16. Denise Rousseau & Byeong Jo Kim & Ryan Splenda & Sarah Young & Jangbum Lee & Donna Beck, 2023. "Does chief executive compensation predict financial performance or inaccurate financial reporting in listed companies: A systematic review," Campbell Systematic Reviews, John Wiley & Sons, vol. 19(4), December.
    17. Surbhi & Sarita Anand, 2019. "Notes From the Field: Using Grassroots Comics to Break the Silence on Menstruation," Indian Journal of Gender Studies, Centre for Women's Development Studies, vol. 26(1-2), pages 171-182, February.
    18. Cigdem Hursen & Derviş Paşa & Hafize Keser, 2023. "High School Students’ Use of Information, Media, and Technology Skills and Multidimensional 21st-Century Skills: An Investigation within the Context of Students, Teachers, and Curricula," Sustainability, MDPI, vol. 15(16), pages 1-21, August.
    19. Gkinko, Lorentsa & Elbanna, Amany, 2023. "Designing trust: The formation of employees’ trust in conversational AI in the digital workplace," Journal of Business Research, Elsevier, vol. 158(C).
    20. Liu, Yu-li & Song, Chuling & Huang, Dongpeng & Liu, Sunny Xun & Li, Zhuoyang & Hu, Bo, 2023. "How can chatbots effectively comfort humans? Exploring the interaction of emoji and identity cues for perceived emotional support," 32nd European Regional ITS Conference, Madrid 2023: Realising the digital decade in the European Union – Easier said than done? 277999, International Telecommunications Society (ITS).

    More about this item

    Keywords

    Artificial intelligence assistants; Conversational agents; Chatbots; Quality criteria set; Design science research (DSR);
    All these keywords.

    JEL classification:

    • M15 - Business Administration and Business Economics; Marketing; Accounting; Personnel Economics - - Business Administration - - - IT Management

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:elmark:v:33:y:2023:i:1:d:10.1007_s12525-023-00662-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.