IDEAS home Printed from https://ideas.repec.org/a/gam/jftint/v17y2025i2p64-d1583830.html
   My bibliography  Save this article

A Weighted Composite Metric for Evaluating User Experience in Educational Chatbots: Balancing Usability, Engagement, and Effectiveness

Author

Listed:
  • Abeer Alabbas

    (Department of Technical Support, Faculty of Technical College for Girls, Technical and Vocational Training Corporation, Najran 66253, Saudi Arabia)

  • Khalid Alomar

    (Department of Information Systems, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah 21589, Saudi Arabia)

Abstract

Evaluating user experience (UX) is essential for optimizing educational chatbots to enhance learning outcomes and student productivity. This study introduces a novel weighted composite metric integrating interface usability assessment (via the Chatbot Usability Questionnaire, CUQ), engagement measurements (via the User Engagement Scale—Short Form, UES-SF), and objective performance indicators (through error rates and response times), addressing gaps in existing evaluation methods across interaction modes (text-based, menu-based, and hybrid) and question complexities. A 3 × 3 within-subject experimental design ( n = 30) was conducted, measuring these distinct UX dimensions through standardized instruments and performance metrics, supplemented by qualitative feedback. Principal Component Analysis (PCA) was used to derive weights for the composite UX metric based on empirical patterns in user interactions. Repeated-measures ANOVA revealed that the hybrid interaction mode outperformed the others, achieving significantly higher usability (F(2,58) = 89.32, p < 0.001) and engagement (F(2,58) = 8.67, p < 0.001), with fewer errors and faster response times under complex query conditions. These findings demonstrate the hybrid mode’s adaptability across question complexities. The proposed framework establishes a standardized method for evaluating educational chatbots, providing actionable insights for interface optimization and sustainable learning tools.

Suggested Citation

  • Abeer Alabbas & Khalid Alomar, 2025. "A Weighted Composite Metric for Evaluating User Experience in Educational Chatbots: Balancing Usability, Engagement, and Effectiveness," Future Internet, MDPI, vol. 17(2), pages 1-35, February.
    • Handle: RePEc:gam:jftint:v:17:y:2025:i:2:p:64-:d:1583830
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1999-5903/17/2/64/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1999-5903/17/2/64/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Heather L. O'Brien & Elaine G. Toms, 2008. "What is user engagement? A conceptual framework for defining user engagement with technology," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(6), pages 938-955, April.
    2. Humaid Al Naqbi & Zied Bahroun & Vian Ahmed, 2024. "Enhancing Work Productivity through Generative Artificial Intelligence: A Comprehensive Literature Review," Sustainability, MDPI, vol. 16(3), pages 1-37, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:ers:journl:v:xxiv:y:2021:i:2b:p:515-537 is not listed on IDEAS
    2. Daniel Steele & Edda Bild & Cynthia Tarlao & Catherine Guastavino, 2019. "Soundtracking the Public Space: Outcomes of the Musikiosk Soundscape Intervention," IJERPH, MDPI, vol. 16(10), pages 1-38, May.
    3. Yuntao Guo & Srinivas Peeta & Shubham Agrawal & Irina Benedyk, 2022. "Impacts of Pokémon GO on route and mode choice decisions: exploring the potential for integrating augmented reality, gamification, and social components in mobile apps to influence travel decisions," Transportation, Springer, vol. 49(2), pages 395-444, April.
    4. Mariani, Marcello M. & Hashemi, Novin & Wirtz, Jochen, 2023. "Artificial intelligence empowered conversational agents: A systematic literature review and research agenda," Journal of Business Research, Elsevier, vol. 161(C).
    5. Nripesh Trivedi & Daniel Adomako Asamoah & Derek Doran, 2018. "Keep the conversations going: engagement-based customer segmentation on online social service platforms," Information Systems Frontiers, Springer, vol. 20(2), pages 239-257, April.
    6. Julia Troll & Ivo Blohm & Jan Marco Leimeister, 2019. "Why Incorporating a Platform-Intermediary can Increase Crowdsourcees’ Engagement," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 61(4), pages 433-450, August.
    7. Matteo Bodini, 2024. "Generative Artificial Intelligence and Regulations: Can We Plan a Resilient Journey Toward the Safe Application of Generative Artificial Intelligence?," Societies, MDPI, vol. 14(12), pages 1-16, December.
    8. Mariana Cernicova-Buca & Adina Palea, 2021. "An Appraisal of Communication Practices Demonstrated by Romanian District Public Health Authorities at the Outbreak of the COVID-19 Pandemic," Sustainability, MDPI, vol. 13(5), pages 1-19, February.
    9. Afi Fouad El & Ouiddad Smail, 2021. "Consumer engagement in value co-creation within virtual video game communities," Management & Marketing, Sciendo, vol. 16(4), pages 370-386, December.
    10. Luca Longo, 2018. "Experienced mental workload, perception of usability, their interaction and impact on task performance," PLOS ONE, Public Library of Science, vol. 13(8), pages 1-36, August.
    11. Zixing Shen & Michael J. Pritchard, 2022. "Cognitive engagement on social media: A study of the effects of visual cueing in educational videos," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 73(9), pages 1253-1267, September.
    12. Shahbaznezhad, Hamidreza & Dolan, Rebecca & Rashidirad, Mona, 2021. "The Role of Social Media Content Format and Platform in Users' Engagement Behavior," Journal of Interactive Marketing, Elsevier, vol. 53(C), pages 47-65.
    13. Fang, Jiaming & Zhao, Zhirong & Wen, Chao & Wang, Ruping, 2017. "Design and performance attributes driving mobile travel application engagement," International Journal of Information Management, Elsevier, vol. 37(4), pages 269-283.
    14. Jezdancher Watti & Máté Millner & Kata Siklósi & Hedvig Kiss & Oguz Kelemen & Dávid Pócs, 2022. "Smokers’ Engagement Behavior on Facebook: Verbalizing and Visual Expressing the Smoking Cessation Process," IJERPH, MDPI, vol. 19(16), pages 1-14, August.
    15. Jian Zhang & Fang Liu, 2024. "Second Screen Synergy: Exploring the Socio-Economic Impact of Dual Screen Engagement in Television Consumption," Journal of the Knowledge Economy, Springer;Portland International Center for Management of Engineering and Technology (PICMET), vol. 15(4), pages 20196-20228, December.
    16. Lin Shi & Katharine J. Mach & Sangwon Suh & Adam Brandt, 2022. "Functionality‐based life cycle assessment framework: An information and communication technologies (ICT) product case study," Journal of Industrial Ecology, Yale University, vol. 26(3), pages 782-800, June.
    17. Wenrui Wang, 2023. "The Ways that Digital Technologies Inform Visitor's Engagement with Cultural Heritage Sites: Informal Learning in the Digital Era ," GATR Journals gjbssr628, Global Academy of Training and Research (GATR) Enterprise.
    18. Caitríona Ní Shé & Eabhnat Ní Fhloinn & Ciarán Mac an Bhaird, 2023. "Student Engagement with Technology-Enhanced Resources in Mathematics in Higher Education: A Review," Mathematics, MDPI, vol. 11(3), pages 1-34, February.
    19. Heather L. O'Brien, 2017. "Antecedents and learning outcomes of online news engagement," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(12), pages 2809-2820, December.
    20. Hwang, Angel Hsing-Chi & Oh, Jeeyun, 2020. "Interacting with background music engages E-Customers more: The impact of interactive music on consumer perception and behavioral intention," Journal of Retailing and Consumer Services, Elsevier, vol. 54(C).
    21. Bitrián, Paula & Buil, Isabel & Catalán, Sara, 2021. "Enhancing user engagement: The role of gamification in mobile apps," Journal of Business Research, Elsevier, vol. 132(C), pages 170-185.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jftint:v:17:y:2025:i:2:p:64-:d:1583830. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.