IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2407.17624.html
   My bibliography  Save this paper

Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMs

Author

Listed:
  • Felix Drinkall
  • Janet B. Pierrehumbert
  • Stefan Zohren

Abstract

Large Language Models (LLMs) have been shown to perform well for many downstream tasks. Transfer learning can enable LLMs to acquire skills that were not targeted during pre-training. In financial contexts, LLMs can sometimes beat well-established benchmarks. This paper investigates how well LLMs perform in the task of forecasting corporate credit ratings. We show that while LLMs are very good at encoding textual information, traditional methods are still very competitive when it comes to encoding numeric and multimodal data. For our task, current LLMs perform worse than a more traditional XGBoost architecture that combines fundamental and macroeconomic data with high-density text-based embedding features.

Suggested Citation

  • Felix Drinkall & Janet B. Pierrehumbert & Stefan Zohren, 2024. "Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMs," Papers 2407.17624, arXiv.org, revised Jan 2025.
    • Handle: RePEc:arx:papers:2407.17624
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2407.17624
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Taylor, Mark P. & Wang, Zigan & Xu, Qi, 2021. "The real effects of exchange rate risk on corporate investment: International evidence," Journal of International Money and Finance, Elsevier, vol. 117(C).
    2. Carling, Kenneth & Jacobson, Tor & Linde, Jesper & Roszbach, Kasper, 2007. "Corporate credit risk modeling and the macroeconomy," Journal of Banking & Finance, Elsevier, vol. 31(3), pages 845-868, March.
    3. Eleni Kalamara & Arthur Turrell & Chris Redl & George Kapetanios & Sujit Kapadia, 2022. "Making text count: Economic forecasting using newspaper text," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(5), pages 896-919, August.
    4. Luca Barbaglia & Sergio Consoli & Sebastiano Manzan, 2023. "Forecasting with Economic News," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 41(3), pages 708-719, July.
    5. Wuyue An & Lin Wang & Yu‐Rong Zeng, 2023. "Text‐based soybean futures price forecasting: A two‐stage deep learning approach," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(2), pages 312-330, March.
    6. Mai, Feng & Tian, Shaonan & Lee, Chihoon & Ma, Ling, 2019. "Deep learning models for bankruptcy prediction using textual disclosures," European Journal of Operational Research, Elsevier, vol. 274(2), pages 743-758.
    7. Romain Boulland & François Degeorge & Edith Ginglinger, 2017. "News Dissemination and Investor Attention," Review of Finance, European Finance Association, vol. 21(2), pages 761-791.
    8. Xinli Yu & Zheng Chen & Yuan Ling & Shujing Dong & Zongyi Liu & Yanbin Lu, 2023. "Temporal Data Meets LLM -- Explainable Financial Time Series Forecasting," Papers 2306.11025, arXiv.org.
    9. Mahsa Tavakoli & Rohitash Chandra & Fengrui Tian & Cristi'an Bravo, 2023. "Multi-Modal Deep Learning for Credit Rating Prediction Using Text and Numerical Data Streams," Papers 2304.10740, arXiv.org, revised Nov 2024.
    10. Hyungjin Cho & Seung-Youb Han & Seungbin Oh & Hee-Yeon Sunwoo, 2020. "Optimistic credit rating and its influence on corporate decisions: evidence from Korea," Asia-Pacific Journal of Accounting & Economics, Taylor & Francis Journals, vol. 27(5), pages 612-629, September.
    11. Stevenson, Matthew & Mues, Christophe & Bravo, Cristián, 2021. "The value of text for small business default prediction: A Deep Learning approach," European Journal of Operational Research, Elsevier, vol. 295(2), pages 758-771.
    12. Baresa, Suzana & Bogdan, Sinisa & Ivanovic, Sasa, 2012. "Role, Interests And Critics Of Credit Rating Agencies," UTMS Journal of Economics, University of Tourism and Management, Skopje, Macedonia, vol. 3(1), pages 71-82.
    13. Yuqi Nie & Yaxuan Kong & Xiaowen Dong & John M. Mulvey & H. Vincent Poor & Qingsong Wen & Stefan Zohren, 2024. "A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges," Papers 2406.11903, arXiv.org.
    14. Xi Zhang & Yunjia Zhang & Senzhang Wang & Yuntao Yao & Binxing Fang & Philip S. Yu, 2018. "Improving Stock Market Prediction via Heterogeneous Information Fusion," Papers 1801.00588, arXiv.org.
    15. Tim Loughran & Bill Mcdonald, 2011. "When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10‐Ks," Journal of Finance, American Finance Association, vol. 66(1), pages 35-65, February.
    16. Yupeng Cao & Zhi Chen & Prashant Kumar & Qingyun Pei & Yangyang Yu & Haohang Li & Fabrizio Dimino & Lorenzo Ausiello & K. P. Subbalakshmi & Papa Momar Ndiaye, 2024. "RiskLabs: Predicting Financial Risk Using Large Language Model based on Multimodal and Multi-Sources Data," Papers 2404.07452, arXiv.org, revised May 2025.
    17. Minchae Song & Kyung‐shik Shin, 2019. "Forecasting economic indicators using a consumer sentiment index: Survey‐based versus text‐based data," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 38(6), pages 504-518, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Beckert, Bernd & Kroll, Henning, 2024. "Definition of the research and innovation field of "Artificial Intelligence" and approaches to determining quality," Discussion Papers "Innovation Systems and Policy Analysis" 88, Fraunhofer Institute for Systems and Innovation Research (ISI).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Luca Barbaglia & Sergio Consoli & Sebastiano Manzan, 2024. "Forecasting GDP in Europe with textual data," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 39(2), pages 338-355, March.
    2. Christopher Gerling & Stefan Lessmann, 2023. "Multimodal Document Analytics for Banking Process Automation," Papers 2307.11845, arXiv.org, revised Nov 2023.
    3. Kriebel, Johannes & Stitz, Lennart, 2022. "Credit default prediction from user-generated text in peer-to-peer lending using deep learning," European Journal of Operational Research, Elsevier, vol. 302(1), pages 309-323.
    4. Katsafados, Apostolos G. & Leledakis, George N. & Pyrgiotakis, Emmanouil G. & Androutsopoulos, Ion & Fergadiotis, Manos, 2024. "Machine learning in bank merger prediction: A text-based approach," European Journal of Operational Research, Elsevier, vol. 312(2), pages 783-797.
    5. Julian Ashwin & Eleni Kalamara & Lorena Saiz, 2024. "Nowcasting Euro area GDP with news sentiment: A tale of two crises," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 39(5), pages 887-905, August.
    6. Zhao, Shuping & Xu, Kai & Wang, Zhao & Liang, Changyong & Lu, Wenxing & Chen, Bo, 2022. "Financial distress prediction by combining sentiment tone features," Economic Modelling, Elsevier, vol. 106(C).
    7. Martin Baumgaertner & Johannes Zahner, 2021. "Whatever it takes to understand a central banker - Embedding their words using neural networks," MAGKS Papers on Economics 202130, Philipps-Universität Marburg, Faculty of Business Administration and Economics, Department of Economics (Volkswirtschaftliche Abteilung).
    8. Huang, Shirley Hsueh-Li & Hu, Guo-Hsin & Hsu, Ming-Fu, 2025. "Identifying contextual content-based risk drivers for advanced risk management strategies," Research in International Business and Finance, Elsevier, vol. 73(PB).
    9. Das, Ronnie & Ahmed, Wasim & Sharma, Kshitij & Hardey, Mariann & Dwivedi, Yogesh K. & Zhang, Ziqi & Apostolidis, Chrysostomos & Filieri, Raffaele, 2024. "Towards the development of an explainable e-commerce fake review index: An attribute analytics approach," European Journal of Operational Research, Elsevier, vol. 317(2), pages 382-400.
    10. Mahsa Tavakoli & Rohitash Chandra & Fengrui Tian & Cristi'an Bravo, 2023. "Multi-Modal Deep Learning for Credit Rating Prediction Using Text and Numerical Data Streams," Papers 2304.10740, arXiv.org, revised Nov 2024.
    11. Mario Sanz-Guerrero & Javier Arroyo, 2024. "Credit Risk Meets Large Language Models: Building a Risk Indicator from Loan Descriptions in P2P Lending," Papers 2401.16458, arXiv.org, revised Mar 2025.
    12. Shuangyan Li & Guangrui Wang & Yongli Luo, 2022. "Tone of language, financial disclosure, and earnings management: a textual analysis of form 20-F," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 8(1), pages 1-24, December.
    13. Stolbov, Mikhail & Shchepeleva, Maria & Karminsky, Alexander, 2022. "When central bank research meets Google search: A sentiment index of global financial stress," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 81(C).
    14. Aprigliano, Valentina & Emiliozzi, Simone & Guaitoli, Gabriele & Luciani, Andrea & Marcucci, Juri & Monteforte, Libero, 2023. "The power of text-based indicators in forecasting Italian economic activity," International Journal of Forecasting, Elsevier, vol. 39(2), pages 791-808.
    15. Jiang, Cuiqing & Lyu, Ximei & Yuan, Yufei & Wang, Zhao & Ding, Yong, 2022. "Mining semantic features in current reports for financial distress prediction: Empirical evidence from unlisted public firms in China," International Journal of Forecasting, Elsevier, vol. 38(3), pages 1086-1099.
    16. Adämmer, Philipp & Prüser, Jan & Schüssler, Rainer A., 2025. "Forecasting macroeconomic tail risk in real time: Do textual data add value?," International Journal of Forecasting, Elsevier, vol. 41(1), pages 307-320.
    17. Shi, Yong & Qu, Yi & Chen, Zhensong & Mi, Yunlong & Wang, Yunong, 2024. "Improved credit risk prediction based on an integrated graph representation learning approach with graph transformation," European Journal of Operational Research, Elsevier, vol. 315(2), pages 786-801.
    18. Barbaglia, Luca & Bellia, Mario & Di Girolamo, Francesca & Rho, Caterina, 2024. "Crypto news and policy innovations: Are European markets affected?," JRC Working Papers in Economics and Finance 2024-07, Joint Research Centre, European Commission.
    19. Anastassia Fedyk, 2024. "Front‐Page News: The Effect of News Positioning on Financial Markets," Journal of Finance, American Finance Association, vol. 79(1), pages 5-33, February.
    20. Arnav Grover, 2025. "FinRLlama: A Solution to LLM-Engineered Signals Challenge at FinRL Contest 2024," Papers 2502.01992, arXiv.org.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2407.17624. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.