IDEAS home Printed from https://ideas.repec.org/a/gam/jftint/v16y2023i1p4-d1306264.html
   My bibliography  Save this article

Development of an Assessment Scale for Measurement of Usability and User Experience Characteristics of Bing Chat Conversational AI

Author

Listed:
  • Goran Bubaš

    (Faculty of Organization and Informatics, University of Zagreb, 42000 Varaždin, Croatia)

  • Antonela Čižmešija

    (Faculty of Organization and Informatics, University of Zagreb, 42000 Varaždin, Croatia)

  • Andreja Kovačić

    (Faculty of Organization and Informatics, University of Zagreb, 42000 Varaždin, Croatia)

Abstract

After the introduction of the ChatGPT conversational artificial intelligence (CAI) tool in November 2022, there has been a rapidly growing interest in the use of such tools in higher education. While the educational uses of some other information technology (IT) tools (including collaboration and communication tools, learning management systems, chatbots, and videoconferencing tools) have been frequently evaluated regarding technology acceptance and usability attributes of those technologies, similar evaluations of CAI tools and services like ChatGPT, Bing Chat, and Bard have only recently started to appear in the scholarly literature. In our study, we present a newly developed set of assessment scales that are related to the usability and user experiences of CAI tools when used by university students, as well as the results of evaluation of these assessment scales specifically regarding the CAI Bing Chat tool (i.e., Microsoft Copilot). The following scales were developed and evaluated using a convenience sample (N = 126) of higher education students: Perceived Usefulness, General Usability, Learnability, System Reliability, Visual Design and Navigation, Information Quality, Information Display, Cognitive Involvement, Design Appeal, Trust, Personification, Risk Perception, and Intention to Use. For most of the aforementioned scales, internal consistency (Cronbach alpha) was in the range from satisfactory to good, which implies their potential usefulness for further studies of related attributes of CAI tools. A stepwise linear regression revealed that the most influential predictors of Intention to Use Bing Chat (or ChatGPT) in the future were the usability variable Perceived Usefulness and two user experience variables—Trust and Design Appeal. Also, our study revealed that students’ perceptions of various specific usability and user experience characteristics of Bing Chat were predominantly positive. The evaluated assessment scales could be beneficial in further research that would include other CAI tools like ChatGPT/GPT-4 and Bard.

Suggested Citation

  • Goran Bubaš & Antonela Čižmešija & Andreja Kovačić, 2023. "Development of an Assessment Scale for Measurement of Usability and User Experience Characteristics of Bing Chat Conversational AI," Future Internet, MDPI, vol. 16(1), pages 1-19, December.
    • Handle: RePEc:gam:jftint:v:16:y:2023:i:1:p:4-:d:1306264
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1999-5903/16/1/4/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1999-5903/16/1/4/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Xinjie Deng & Zhonggen Yu, 2023. "A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education," Sustainability, MDPI, vol. 15(4), pages 1-19, February.
    2. Foroughi, Behzad & Nhan, Pham Viet & Iranmanesh, Mohammad & Ghobakhloo, Morteza & Nilashi, Mehrbakhsh & Yadegaridehkordi, Elaheh, 2023. "Determinants of intention to use autonomous vehicles: Findings from PLS-SEM and ANFIS," Journal of Retailing and Consumer Services, Elsevier, vol. 70(C).
    3. Jörg Garrel & Jana Mayer, 2023. "Artificial Intelligence in studies—use of ChatGPT and AI-based tools among students in Germany," Palgrave Communications, Palgrave Macmillan, vol. 10(1), pages 1-9, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Alexandros Tassios & Stergios Tegos & Christos Bouas & Konstantinos Manousaridis & Maria Papoutsoglou & Maria Kaltsa & Eleni Dimopoulou & Thanassis Mavropoulos & Stefanos Vrochidis & Georgios Meditsko, 2025. "LLM Performance in Low-Resource Languages: Selecting an Optimal Model for Migrant Integration Support in Greek," Future Internet, MDPI, vol. 17(6), pages 1-24, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tsai, Pei-Hsuan & Tang, Jia-Wei & Ou, Mei-Ling, 2024. "Applying the extended UTAUT-2 to assess the factors contributing to consumers’ usage intention towards over-the-top video streaming platforms," Journal of Retailing and Consumer Services, Elsevier, vol. 81(C).
    2. Aslan, Marjan & Keating, Byron W., 2025. "Providing social support in technology-based service encounters: Activating intrinsic motivations to create better outcomes," Journal of Retailing and Consumer Services, Elsevier, vol. 84(C).
    3. Soren, Anup Anurag & Chakraborty, Shibashish, 2024. "Adoption, satisfaction, trust, and commitment of over-the-top platforms: An integrated approach," Journal of Retailing and Consumer Services, Elsevier, vol. 76(C).
    4. Dang, Simon & Quach, Sara & Roberts, Robin E., 2025. "Explanation of time perspectives in adopting AI service robots under different service settings," Journal of Retailing and Consumer Services, Elsevier, vol. 82(C).
    5. Jibon Naher, 2023. "A Survey to Understand the experience of ChatGPT Usage among Engineering University Students in Bangladesh," Technium, Technium Science, vol. 15(1), pages 1-8.
    6. Janik Ole Wecks & Johannes Voshaar & Benedikt Jost Plate & Jochen Zimmermann, 2024. "Generative AI Usage and Exam Performance," Papers 2404.19699, arXiv.org, revised Nov 2024.
    7. Pham, Hong Chuong & Duong, Cong Doanh & Nguyen, Giang Khanh Huyen, 2024. "What drives tourists’ continuance intention to use ChatGPT for travel services? A stimulus-organism-response perspective," Journal of Retailing and Consumer Services, Elsevier, vol. 78(C).
    8. Banda, Lakshmi Ganesh & Chandra, Shalini, 2025. "Natural intelligence in autonomous vehicles: Redefining the path to human-centric adoption," Journal of Retailing and Consumer Services, Elsevier, vol. 84(C).
    9. Ajda Fošner, 2024. "University Students’ Attitudes and Perceptions towards AI Tools: Implications for Sustainable Educational Practices," Sustainability, MDPI, vol. 16(19), pages 1-15, October.
    10. Suhail, Faisal & Adel, Mouhand & Al-Emran, Mostafa & AlQudah, Adi Ahmad, 2024. "Are students ready for robots in higher education? Examining the adoption of robots by integrating UTAUT2 and TTF using a hybrid SEM-ANN approach," Technology in Society, Elsevier, vol. 77(C).
    11. Hu, Qian & Pan, Zhao, 2023. "Can AI benefit individual resilience? The mediation roles of AI routinization and infusion," Journal of Retailing and Consumer Services, Elsevier, vol. 73(C).
    12. Chunqin Zhang & Zhangbiao Yu & Yi Huang & Mengmeng Wang & Skitmore Martin & Guangnian Xiao & Xi Lu, 2024. "Investigating the influence mechanism of goal‐framing theory on urban residents' green travel behavior," Sustainable Development, John Wiley & Sons, Ltd., vol. 32(5), pages 5144-5160, October.
    13. Kai Guo & Chengyuan Zhan & Xiang Li, 2025. "Factors influencing Chinese college students’ intention to use AIGC: a study based on the UTAUT model," International Journal of System Assurance Engineering and Management, Springer;The Society for Reliability, Engineering Quality and Operations Management (SREQOM),India, and Division of Operation and Maintenance, Lulea University of Technology, Sweden, vol. 16(4), pages 1663-1677, April.
    14. Xing, Yingying & Li, Lun & Sheng, Xiaolu & Zhou, Huiyu & Han, Xiao, 2025. "Understanding public perceptions of autonomous vehicles: A structural model to urban-rural differences and psychological factors," Transport Policy, Elsevier, vol. 165(C), pages 70-84.
    15. Marco Agustín Arbulú Ballesteros & Benicio Gonzalo Acosta Enríquez & Emma Verónica Ramos Farroñán & Hugo Daniel García Juárez & Luis Edgardo Cruz Salinas & Julio Ernesto Blas Sánchez & Julie Catherine, 2024. "The Sustainable Integration of AI in Higher Education: Analyzing ChatGPT Acceptance Factors Through an Extended UTAUT2 Framework in Peruvian Universities," Sustainability, MDPI, vol. 16(23), pages 1-28, December.
    16. Maria Karyotaki & Athanasios Drigas & Charalabos Skianis, 2024. "Contributions of the 9-Layered Model of Giftedness to the Development of a Conversational Agent for Healthy Ageing and Sustainable Living," Sustainability, MDPI, vol. 16(7), pages 1-18, March.
    17. Vikas Chauhan & Rohit Joshi & Vipin Choudhary, 2024. "Understanding intention to adopt telematics-based automobile insurance in an emerging economy: a mixed-method approach," Journal of Financial Services Marketing, Palgrave Macmillan, vol. 29(3), pages 1017-1036, September.
    18. Crisafulli, B. & Guesalaga, R. & Dimitriu, R., 2025. "Consumers’ adoption of autonomous cars as a personal values-directed behavior," Journal of Business Research, Elsevier, vol. 189(C).
    19. Athanasios Polyportis & Nikolaos Pahos, 2024. "Navigating the perils of artificial intelligence: a focused review on ChatGPT and responsible research and innovation," Palgrave Communications, Palgrave Macmillan, vol. 11(1), pages 1-10, December.
    20. Koh, Le Yi & Yuen, Kum Fai, 2024. "The role of motivators, barriers, attractiveness, and positive emotions on consumers’ intention to adopt and resist self-driving delivery robots," Journal of Retailing and Consumer Services, Elsevier, vol. 81(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jftint:v:16:y:2023:i:1:p:4-:d:1306264. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.