IDEAS home Printed from https://ideas.repec.org/a/gam/jftint/v17y2025i2p64-d1583830.html

A Weighted Composite Metric for Evaluating User Experience in Educational Chatbots: Balancing Usability, Engagement, and Effectiveness

Author

Listed:
  • Abeer Alabbas

    (Department of Technical Support, Faculty of Technical College for Girls, Technical and Vocational Training Corporation, Najran 66253, Saudi Arabia)

  • Khalid Alomar

    (Department of Information Systems, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah 21589, Saudi Arabia)

Abstract

Evaluating user experience (UX) is essential for optimizing educational chatbots to enhance learning outcomes and student productivity. This study introduces a novel weighted composite metric integrating interface usability assessment (via the Chatbot Usability Questionnaire, CUQ), engagement measurements (via the User Engagement Scale—Short Form, UES-SF), and objective performance indicators (through error rates and response times), addressing gaps in existing evaluation methods across interaction modes (text-based, menu-based, and hybrid) and question complexities. A 3 × 3 within-subject experimental design ( n = 30) was conducted, measuring these distinct UX dimensions through standardized instruments and performance metrics, supplemented by qualitative feedback. Principal Component Analysis (PCA) was used to derive weights for the composite UX metric based on empirical patterns in user interactions. Repeated-measures ANOVA revealed that the hybrid interaction mode outperformed the others, achieving significantly higher usability (F(2,58) = 89.32, p < 0.001) and engagement (F(2,58) = 8.67, p < 0.001), with fewer errors and faster response times under complex query conditions. These findings demonstrate the hybrid mode’s adaptability across question complexities. The proposed framework establishes a standardized method for evaluating educational chatbots, providing actionable insights for interface optimization and sustainable learning tools.

Suggested Citation

  • Abeer Alabbas & Khalid Alomar, 2025. "A Weighted Composite Metric for Evaluating User Experience in Educational Chatbots: Balancing Usability, Engagement, and Effectiveness," Future Internet, MDPI, vol. 17(2), pages 1-35, February.
    • Handle: RePEc:gam:jftint:v:17:y:2025:i:2:p:64-:d:1583830
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1999-5903/17/2/64/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1999-5903/17/2/64/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Heather L. O'Brien & Elaine G. Toms, 2008. "What is user engagement? A conceptual framework for defining user engagement with technology," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(6), pages 938-955, April.
    2. Humaid Al Naqbi & Zied Bahroun & Vian Ahmed, 2024. "Enhancing Work Productivity through Generative Artificial Intelligence: A Comprehensive Literature Review," Sustainability, MDPI, vol. 16(3), pages 1-37, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Antonella Pireddu & Claudia Giliberti & Alessandro Innocenti & Carla Simeoni & Michela Bonafede, 2025. "A Novel Macro-Level Model in Evaluating Health and Safety Training Based on Virtual Reality," IJERPH, MDPI, vol. 22(9), pages 1-26, September.
    2. Dalal Al Faia & Khalid Alomar, 2025. "Evaluating Learning Success, Engagement, and Usability of Moalemy: An Arabic Rule-Based Chatbot," Future Internet, MDPI, vol. 17(10), pages 1-35, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nicola Whitton & Alex Moseley, 2014. "Deconstructing Engagement," Simulation & Gaming, , vol. 45(4-5), pages 433-449, August.
    2. Chatterjee, Sidharta, 2025. "Productivity and Productive Capital: Metaphysical Perspectives," MPRA Paper 125316, University Library of Munich, Germany.
    3. M. Eshaq Stanikzai & Ella Mittal, 2025. "Leveraging AI-generated and human-generated content for maximized user engagement in contentpreneurs’ innovation and creativity," Journal of Innovation and Entrepreneurship, Springer, vol. 14(1), pages 1-28, December.
    4. Mariani, Marcello M. & Hashemi, Novin & Wirtz, Jochen, 2023. "Artificial intelligence empowered conversational agents: A systematic literature review and research agenda," Journal of Business Research, Elsevier, vol. 161(C).
    5. Mariana Cernicova-Buca & Adina Palea, 2021. "An Appraisal of Communication Practices Demonstrated by Romanian District Public Health Authorities at the Outbreak of the COVID-19 Pandemic," Sustainability, MDPI, vol. 13(5), pages 1-19, February.
    6. Afi Fouad El & Ouiddad Smail, 2021. "Consumer engagement in value co-creation within virtual video game communities," Management & Marketing, Sciendo, vol. 16(4), pages 370-386, December.
    7. Luca Longo, 2018. "Experienced mental workload, perception of usability, their interaction and impact on task performance," PLOS ONE, Public Library of Science, vol. 13(8), pages 1-36, August.
    8. Shahbaznezhad, Hamidreza & Dolan, Rebecca & Rashidirad, Mona, 2021. "The Role of Social Media Content Format and Platform in Users' Engagement Behavior," Journal of Interactive Marketing, Elsevier, vol. 53(C), pages 47-65.
    9. Varnita Dubey, 2026. "Generative AI in the Workplace: A Systematic Review of Productivity Effects, Employment Perceptions, and Job Insecurity," International Journal of Research and Innovation in Social Science, International Journal of Research and Innovation in Social Science (IJRISS), vol. 10(3), pages 4426-4437, March.
    10. Jezdancher Watti & Máté Millner & Kata Siklósi & Hedvig Kiss & Oguz Kelemen & Dávid Pócs, 2022. "Smokers’ Engagement Behavior on Facebook: Verbalizing and Visual Expressing the Smoking Cessation Process," IJERPH, MDPI, vol. 19(16), pages 1-14, August.
    11. Hwang, Angel Hsing-Chi & Oh, Jeeyun, 2020. "Interacting with background music engages E-Customers more: The impact of interactive music on consumer perception and behavioral intention," Journal of Retailing and Consumer Services, Elsevier, vol. 54(C).
    12. Peters, Twan & Işık, Öykü & Tona, Olgerta & Popovič, Aleš, 2016. "How system quality influences mobile BI use: The mediating role of engagement," International Journal of Information Management, Elsevier, vol. 36(5), pages 773-783.
    13. Linze Li & M. Asyiek Bin Mat Desa & Tiancheng Li & Wenjie Li, 2026. "Overview of the Application of Generative Artificial Intelligence in Film Production: Algorithms, Tools, and Future Trends," Studies in Media and Communication, Redfame publishing, vol. 14(2), pages 22-39, June.
    14. Carolina Herrando & Julio Jiménez-Martínez & María José Martín-De Hoyos, 2017. "Passion at first sight: how to engage users in social commerce contexts," Electronic Commerce Research, Springer, vol. 17(4), pages 701-720, December.
    15. Li, Manning & Mao, Jiye, 2015. "Hedonic or utilitarian? Exploring the impact of communication style alignment on user's perception of virtual health advisory services," International Journal of Information Management, Elsevier, vol. 35(2), pages 229-243.
    16. Alvertos, Konstantinos & Ioannou, Nikos & Kokkinis, Dimitris & Katsianis, Dimitris & Varoutas, Dimitris, 2025. "A Decision-Making Framework for Integrating Generative AI in Enterprise Workflows: Building vs. Buying," 33rd European Regional ITS Conference, Edinburgh, 2025: Digital innovation and transformation in uncertain times 331248, International Telecommunications Society (ITS).
    17. Nripesh Trivedi & Daniel Adomako Asamoah & Derek Doran, 0. "Keep the conversations going: engagement-based customer segmentation on online social service platforms," Information Systems Frontiers, Springer, vol. 0, pages 1-19.
    18. Ayla Schwarz & Greet Cardon & Sebastien Chastin & Jeroen Stragier & Lieven De Marez & Consortium SmartLife & Ann DeSmet, 2021. "Does Dynamic Tailoring of A Narrative-Driven Exergame Result in Higher User Engagement among Adolescents? Results from A Cluster-Randomized Controlled Trial," IJERPH, MDPI, vol. 18(14), pages 1-19, July.
    19. Park, JungKun & Hong, EunPyo & Ahn, Jiseon & Hyun, Hyowon, 2023. "Role of multidimensional customer brand engagement on customer behavior for online grocery shopping," Journal of Retailing and Consumer Services, Elsevier, vol. 73(C).
    20. Pagani, Margherita & Malacarne, Giovanni, 2017. "Experiential Engagement and Active vs. Passive Behavior in Mobile Location-based Social Networks: The Moderating Role of Privacy," Journal of Interactive Marketing, Elsevier, vol. 37(C), pages 133-148.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jftint:v:17:y:2025:i:2:p:64-:d:1583830. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager The email address of this maintainer does not seem to be valid anymore. Please ask MDPI Indexing Manager to update the entry or send us the correct address (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.