IDEAS home Printed from https://ideas.repec.org/a/sae/amerec/v69y2024i1p80-87.html

ChatGPT Hallucinates Non-existent Citations: Evidence from Economics

Author

Listed:
  • Joy Buchanan
  • Stephen Hill
  • Olga Shapoval

Abstract

In this study, we generate prompts derived from every topic within the Journal of Economic Literature to assess the abilities of both GPT-3.5 and GPT-4 versions of the ChatGPT large language model (LLM) to write about economic concepts. ChatGPT demonstrates considerable competency in offering general summaries but also cites non-existent references. More than 30% of the citations provided by the GPT-3.5 version do not exist and this rate is only slightly reduced for the GPT-4 version. Additionally, our findings suggest that the reliability of the model decreases as the prompts become more specific. We provide quantitative evidence for errors in ChatGPT output to demonstrate the importance of LLM verification. JEL Codes: B4; O33; I2

Suggested Citation

  • Joy Buchanan & Stephen Hill & Olga Shapoval, 2024. "ChatGPT Hallucinates Non-existent Citations: Evidence from Economics," The American Economist, Sage Publications, vol. 69(1), pages 80-87, March.
    • Handle: RePEc:sae:amerec:v:69:y:2024:i:1:p:80-87
    DOI: 10.1177/05694345231218454
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/05694345231218454
    Download Restriction: no
    File URL: https://libkey.io/10.1177/05694345231218454?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Wayne Geerling & G. Dirk Mateer & Jadrian Wooten & Nikhil Damodaran, 2023. "ChatGPT has Aced the Test of Understanding in College Economics: Now What?," The American Economist, Sage Publications, vol. 68(2), pages 233-245, October.
    2. Goldfarb, Avi & Taska, Bledi & Teodoridis, Florenta, 2023. "Could machine learning be a general purpose technology? A comparison of emerging technologies using data from online job postings," Research Policy, Elsevier, vol. 52(1).
    3. John J. Horton & Apostolos Filippas & Benjamin S. Manning, 2023. "Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?," NBER Working Papers 31122, National Bureau of Economic Research, Inc.
    4. Tyna Eloundou & Sam Manning & Pamela Mishkin & Daniel Rock, 2023. "GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models," Papers 2303.10130, arXiv.org, revised Aug 2023.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Pawel Gmyrek & Janine Berg & David Bescond, 2025. "Generative AI and Jobs: An Analysis of Potential Effects on Global Employment," Gospodarka Narodowa. The Polish Journal of Economics, Warsaw School of Economics, issue 3, pages 6-30.
    2. Mahyar Habibi, 2025. "Open Sourcing GPTs: Economics of Open Sourcing Advanced AI Models," Papers 2501.11581, arXiv.org.
    3. Frank M. Fossen & Trevor McLemore & Alina Sorgner, 2024. "Artificial Intelligence and Entrepreneurship," Foundations and Trends(R) in Entrepreneurship, now publishers, vol. 20(8), pages 781-904, December.
    4. Piyush Gulati & Arianna Marchetti & Phanish Puranam & Victoria Sevcenko, 2025. "Generative AI Adoption and Higher Order Skills," Papers 2503.09212, arXiv.org, revised Jun 2025.
    5. Kristina McElheran & J. Frank Li & Erik Brynjolfsson & Zachary Kroff & Emin Dinlersoz & Lucia Foster & Nikolas Zolas, 2024. "AI adoption in America: Who, what, and where," Journal of Economics & Management Strategy, Wiley Blackwell, vol. 33(2), pages 375-415, March.
    6. Zenan Chen & Jason Chan, 2024. "Large Language Model in Creative Work: The Role of Collaboration Modality and User Expertise," Management Science, INFORMS, vol. 70(12), pages 9101-9117, December.
    7. Wu, Qinqin & Zhuang, Qinqin & Liu, Yitong & Han, Longyan, 2024. "Technology shock of ChatGPT, social attention and firm value: Evidence from China," Technology in Society, Elsevier, vol. 79(C).
    8. Flavio Calvino & Luca Fontanelli, 2025. "Decoding AI: Nine facts about how firms use artificial intelligence in France," LEM Papers Series 2025/13, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    9. Hui Liang & Jingbo Fan & Yunhan Wang, 2025. "Artificial Intelligence, Technological Innovation, and Employment Transformation for Sustainable Development: Evidence from China," Sustainability, MDPI, vol. 17(9), pages 1-28, April.
    10. Chen, Qin & Ge, Jinfeng & Xie, Huaqing & Xu, Xingcheng & Yang, Yanqing, 2025. "Large language models at work in China’s labor market," China Economic Review, Elsevier, vol. 92(C).
    11. Ritu Agarwal & Michelle Dugas & Guodong (Gordon) Gao, 2024. "Augmenting physicians with artificial intelligence to transform healthcare: Challenges and opportunities," Journal of Economics & Management Strategy, Wiley Blackwell, vol. 33(2), pages 360-374, March.
    12. Benjamin Laufer & Jon Kleinberg & Hoda Heidari, 2023. "Fine-Tuning Games: Bargaining and Adaptation for General-Purpose Models," Papers 2308.04399, arXiv.org, revised Dec 2024.
    13. Bughin, Jacques, 2024. "What drives the corporate payoffs of using generative artificial intelligence?," Structural Change and Economic Dynamics, Elsevier, vol. 71(C), pages 658-668.
    14. Tamay Besiroglu & Nicholas Emery-Xu & Neil Thompson, 2022. "Economic impacts of AI-augmented R&D," Papers 2212.08198, arXiv.org, revised Jan 2023.
    15. Christoph Riedl & Eric Bogert, 2024. "Who Benefits from AI? Self-Selection, Skill Gap, and the Hidden Costs of AI Feedback," Papers 2409.18660, arXiv.org, revised Apr 2026.
    16. Carvajal, Daniel & Franco, Catalina & Isaksson, Siri, 2024. "Will Artificial Intelligence Get in the Way of Achieving Gender Equality?," Discussion Paper Series in Economics 3/2024, Norwegian School of Economics, Department of Economics, revised 28 Apr 2025.
    17. Naomi Hausman & Oren Rigbi & Sarit Weisburd, 2025. "Generative AI’s Impact on Student Achievement and Implications for Worker Productivity," CESifo Working Paper Series 11843, CESifo.
    18. Benedetta Montanaro & Annalisa Croce & Elisa Ughetto, 2024. "Venture capital investments in artificial intelligence," Journal of Evolutionary Economics, Springer, vol. 34(1), pages 1-28, January.
    19. Iñaki Aldasoro & Ajit Desai, 2025. "Money Talks: AI Agents for Cash Management in Payment Systems," Staff Working Papers 25-35, Bank of Canada.
    20. Evangelos Katsamakas & Oleg V. Pavlov & Ryan Saklad, 2024. "Artificial intelligence and the transformation of higher education institutions," Papers 2402.08143, arXiv.org.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    JEL classification:

    • B4 - Schools of Economic Thought and Methodology - - Economic Methodology
    • O33 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights - - - Technological Change: Choices and Consequences; Diffusion Processes
    • I2 - Health, Education, and Welfare - - Education

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:amerec:v:69:y:2024:i:1:p:80-87. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: http://journals.sagepub.com/home/aex .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.