IDEAS home Printed from https://ideas.repec.org/a/gam/jftint/v15y2023i10p336-d1259029.html

Fluent but Not Factual: A Comparative Analysis of ChatGPT and Other AI Chatbots’ Proficiency and Originality in Scientific Writing for Humanities

Author

Listed:
  • Edisa Lozić

    (Research Centre of the Slovenian Academy of Sciences and Arts, 1000 Ljubljana, Slovenia)

  • Benjamin Štular

    (Research Centre of the Slovenian Academy of Sciences and Arts, 1000 Ljubljana, Slovenia)

Abstract

Historically, mastery of writing was deemed essential to human progress. However, recent advances in generative AI have marked an inflection point in this narrative, including for scientific writing. This article provides a comprehensive analysis of the capabilities and limitations of six AI chatbots in scholarly writing in the humanities and archaeology. The methodology was based on tagging AI-generated content for quantitative accuracy and qualitative precision by human experts. Quantitative accuracy assessed the factual correctness in a manner similar to grading students, while qualitative precision gauged the scientific contribution similar to reviewing a scientific article. In the quantitative test, ChatGPT-4 scored near the passing grade (−5) whereas ChatGPT-3.5 (−18), Bing (−21) and Bard (−31) were not far behind. Claude 2 (−75) and Aria (−80) scored much lower. In the qualitative test, all AI chatbots, but especially ChatGPT-4, demonstrated proficiency in recombining existing knowledge, but all failed to generate original scientific content. As a side note, our results suggest that with ChatGPT-4, the size of large language models has reached a plateau. Furthermore, this paper underscores the intricate and recursive nature of human research. This process of transforming raw data into refined knowledge is computationally irreducible, highlighting the challenges AI chatbots face in emulating human originality in scientific writing. Our results apply to the state of affairs in the third quarter of 2023. In conclusion, while large language models have revolutionised content generation, their ability to produce original scientific contributions in the humanities remains limited. We expect this to change in the near future as current large language model-based AI chatbots evolve into large language model-powered software.

Suggested Citation

  • Edisa Lozić & Benjamin Štular, 2023. "Fluent but Not Factual: A Comparative Analysis of ChatGPT and Other AI Chatbots’ Proficiency and Originality in Scientific Writing for Humanities," Future Internet, MDPI, vol. 15(10), pages 1-26, October.
    • Handle: RePEc:gam:jftint:v:15:y:2023:i:10:p:336-:d:1259029
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1999-5903/15/10/336/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1999-5903/15/10/336/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Holly Else, 2023. "Abstracts written by ChatGPT fool scientists," Nature, Nature, vol. 613(7944), pages 423-423, January.
    2. Luc W Nagtegaal & Renger E de Bruin, 1994. "The French connection and other neo-colonial patterns in the global network of science," Research Evaluation, Oxford University Press, vol. 4(2), pages 119-127, August.
    3. Cristòfol Rovira & Lluís Codina & Carlos Lopezosa, 2021. "Language Bias in the Google Scholar Ranking Algorithm," Future Internet, MDPI, vol. 13(2), pages 1-17, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Christopher J. Lynch & Erik J. Jensen & Virginia Zamponi & Kevin O’Brien & Erika Frydenlund & Ross Gore, 2023. "A Structured Narrative Prompt for Prompting Narratives from Large Language Models: Sentiment Assessment of ChatGPT-Generated Narratives and Real Tweets," Future Internet, MDPI, vol. 15(12), pages 1-36, November.
    2. Ketmanto Wangsa & Shakir Karim & Ergun Gide & Mahmoud Elkhodr, 2024. "A Systematic Review and Comprehensive Analysis of Pioneering AI Chatbot Models from Education to Healthcare: ChatGPT, Bard, Llama, Ernie and Grok," Future Internet, MDPI, vol. 16(7), pages 1-23, June.
    3. Sameh Kamal Mohamed Ibrahim & Zakaria Abdelaziz Zakaria Mahmoud, 2026. "Generative AI in academic writing: a comparison of human-authored and ChatGPT-generated research article titles," Humanities and Social Sciences Communications, Palgrave Macmillan, vol. 13(1), pages 1-10, December.
    4. Sakhi Aggrawal & Alejandra J. Magana, 2024. "Teamwork Conflict Management Training and Conflict Resolution Practice via Large Language Models," Future Internet, MDPI, vol. 16(5), pages 1-25, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Arpan Kumar Kar & P. S. Varsha & Shivakami Rajan, 2023. "Unravelling the Impact of Generative Artificial Intelligence (GAI) in Industrial Applications: A Review of Scientific and Grey Literature," Global Journal of Flexible Systems Management, Springer;Global Institute of Flexible Systems Management, vol. 24(4), pages 659-689, December.
    2. Ortega, José Luis & Aguillo, Isidro F., 2013. "Institutional and country collaboration in an online service of scientific profiles: Google Scholar Citations," Journal of Informetrics, Elsevier, vol. 7(2), pages 394-403.
    3. Wang Xian & Chen Guomin & Varsha Arya & Kwok Tai Chui, 2024. "Examining the Influence of AI Chatbots on Semantic Web-Based Global Information Management in Various Industries," International Journal on Semantic Web and Information Systems (IJSWIS), IGI Global Scientific Publishing, vol. 20(1), pages 1-14, January.
    4. Rey-Rocha Jesús & Martín-Sempere María José, 2004. "Patterns of the foreign contributions in some domestic vs. international journals on Earth Sciences," Scientometrics, Springer;Akadémiai Kiadó, vol. 59(1), pages 95-115, January.
    5. Alin ZAMFIROIU & Denisa VASILE & Daniel SAVU, 2023. "ChatGPT – A Systematic Review of Published Research Papers," Informatica Economica, Academy of Economic Studies - Bucharest, Romania, vol. 27(1), pages 5-16.
    6. Raghu Raman & Hiran H Lathabai & Santanu Mandal & Payel Das & Tavleen Kaur & Prema Nedungadi, 2024. "ChatGPT: Literate or intelligent about UN sustainable development goals?," PLOS ONE, Public Library of Science, vol. 19(4), pages 1-27, April.
    7. Guido Schryen & Mauricio Marrone & Jiaqi Yang, 2025. "Exploring the scope of generative AI in literature review development," Electronic Markets, Springer;IIM University of St. Gallen, vol. 35(1), pages 1-26, December.
    8. Wolfgang Glänzel & Lin Zhang, 2018. "Scientometric research assessment in the developing world: A tribute to Michael J. Moravcsik from the perspective of the twenty-first century," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(3), pages 1517-1532, June.
    9. Amina Badreddine & Hadjira Larbi Cherif, 2023. "ChatGPT in Academic Research: Demonstrating Limitations through Real Practical Examples," Post-Print hal-04379581, HAL.
    10. Howell, Bronwyn E. & Potgieter, Petrus H., 2023. "AI-generated lemons: a sour outlook for content producers?," 32nd European Regional ITS Conference, Madrid 2023: Realising the digital decade in the European Union – Easier said than done? 277971, International Telecommunications Society (ITS).
    11. Xinhe Zhang & Tianshu Gu & Sidharth Loganathan & Yanjun Xie & Jinghong Chen, 2025. "Retracted articles on stem cells are continuously cited in publications and used by ChatGPT," Scientometrics, Springer;Akadémiai Kiadó, vol. 130(11), pages 6503-6511, November.
    12. Li-Yuan Huang & Xun Zhang & Qiang Wang & Zhen-Song Chen & Yang Liu, 2024. "Evaluating Media Knowledge Capabilities of Intelligent Search Dialogue Systems: A Case Study of ChatGPT and New Bing," Journal of the Knowledge Economy, Springer;Portland International Center for Management of Engineering and Technology (PICMET), vol. 15(4), pages 17284-17307, December.
    13. Borker, Girija, 2024. "Understanding the constraints to women’s use of urban public transport in developing countries," World Development, Elsevier, vol. 180(C).
    14. Wang, Wenyang & Luo, Yuping & Ma, Mingrui & Wang, Jinglin & Sui, Cong, 2025. "A novel forecasting framework leveraging large language model and machine learning for methanol price," Energy, Elsevier, vol. 320(C).
    15. Konstantinos I. Roumeliotis & Nikolaos D. Tselikas & Dimitrios K. Nasiopoulos, 2022. "Airlines’ Sustainability Study Based on Search Engine Optimization Techniques and Technologies," Sustainability, MDPI, vol. 14(18), pages 1-23, September.
    16. Cong William Lin & Wu Zhu, 2025. "Divergent LLM Adoption and Heterogeneous Convergence Paths in Research Writing," Papers 2504.13629, arXiv.org.
    17. Fiedler, Alexandra & Döpke, Jörg, 2025. "Do humans identify AI-generated text better than machines? Evidence based on excerpts from German theses☆," International Review of Economics Education, Elsevier, vol. 49(C).
    18. Paweł Gmyrek & Christoph Lutz & Gemma Newlands, 2025. "A technological construction of society: Comparing GPT‐4 and human respondents for occupational evaluation in the UK," British Journal of Industrial Relations, London School of Economics, vol. 63(1), pages 180-208, March.
    19. François-Xavier de Vaujany & Aurélie Leclercq Vandelannoitte & Jeremy Aroles & Lucas Introna & Scott Davidson, 2025. "Rethinking responsibility in the digital age: a narrative approach," Post-Print hal-04962366, HAL.
    20. W. Glänzel & A. Schubert & H. -J. Czerwon, 1999. "A bibliometric analysis of international scientific cooperation of the European Union (1985–1995)," Scientometrics, Springer;Akadémiai Kiadó, vol. 45(2), pages 185-202, June.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jftint:v:15:y:2023:i:10:p:336-:d:1259029. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager The email address of this maintainer does not seem to be valid anymore. Please ask MDPI Indexing Manager to update the entry or send us the correct address (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.