IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2407.17624.html
   My bibliography  Save this paper

Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMs

Author

Listed:
  • Felix Drinkall
  • Janet B. Pierrehumbert
  • Stefan Zohren

Abstract

Large Language Models (LLMs) have been shown to perform well for many downstream tasks. Transfer learning can enable LLMs to acquire skills that were not targeted during pre-training. In financial contexts, LLMs can sometimes beat well-established benchmarks. This paper investigates how well LLMs perform in the task of forecasting corporate credit ratings. We show that while LLMs are very good at encoding textual information, traditional methods are still very competitive when it comes to encoding numeric and multimodal data. For our task, current LLMs perform worse than a more traditional XGBoost architecture that combines fundamental and macroeconomic data with high-density text-based embedding features.

Suggested Citation

  • Felix Drinkall & Janet B. Pierrehumbert & Stefan Zohren, 2024. "Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMs," Papers 2407.17624, arXiv.org, revised Jan 2025.
    • Handle: RePEc:arx:papers:2407.17624
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2407.17624
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Taylor, Mark P. & Wang, Zigan & Xu, Qi, 2021. "The real effects of exchange rate risk on corporate investment: International evidence," Journal of International Money and Finance, Elsevier, vol. 117(C).
    2. Carling, Kenneth & Jacobson, Tor & Linde, Jesper & Roszbach, Kasper, 2007. "Corporate credit risk modeling and the macroeconomy," Journal of Banking & Finance, Elsevier, vol. 31(3), pages 845-868, March.
    3. Eleni Kalamara & Arthur Turrell & Chris Redl & George Kapetanios & Sujit Kapadia, 2020. "Making text count: economic forecasting using newspaper text," Bank of England working papers 865, Bank of England.
    4. Luca Barbaglia & Sergio Consoli & Sebastiano Manzan, 2023. "Forecasting with Economic News," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 41(3), pages 708-719, July.
    5. Wuyue An & Lin Wang & Yu‐Rong Zeng, 2023. "Text‐based soybean futures price forecasting: A two‐stage deep learning approach," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(2), pages 312-330, March.
    6. Mai, Feng & Tian, Shaonan & Lee, Chihoon & Ma, Ling, 2019. "Deep learning models for bankruptcy prediction using textual disclosures," European Journal of Operational Research, Elsevier, vol. 274(2), pages 743-758.
    7. Romain Boulland & François Degeorge & Edith Ginglinger, 2017. "News Dissemination and Investor Attention," Review of Finance, European Finance Association, vol. 21(2), pages 761-791.
    8. Xinli Yu & Zheng Chen & Yuan Ling & Shujing Dong & Zongyi Liu & Yanbin Lu, 2023. "Temporal Data Meets LLM -- Explainable Financial Time Series Forecasting," Papers 2306.11025, arXiv.org.
    9. Mahsa Tavakoli & Rohitash Chandra & Fengrui Tian & Cristi'an Bravo, 2023. "Multi-Modal Deep Learning for Credit Rating Prediction Using Text and Numerical Data Streams," Papers 2304.10740, arXiv.org, revised Nov 2024.
    10. Hyungjin Cho & Seung-Youb Han & Seungbin Oh & Hee-Yeon Sunwoo, 2020. "Optimistic credit rating and its influence on corporate decisions: evidence from Korea," Asia-Pacific Journal of Accounting & Economics, Taylor & Francis Journals, vol. 27(5), pages 612-629, September.
    11. Stevenson, Matthew & Mues, Christophe & Bravo, Cristián, 2021. "The value of text for small business default prediction: A Deep Learning approach," European Journal of Operational Research, Elsevier, vol. 295(2), pages 758-771.
    12. Suzana Baresa & Sinisa Bogdan & Sasa Ivanovic, 2012. "Role, Interests And Critics Of Credit Rating Agencies," UTMS Journal of Economics, University of Tourism and Management, Skopje, Macedonia, vol. 3(1), pages 71-82.
    13. Yuqi Nie & Yaxuan Kong & Xiaowen Dong & John M. Mulvey & H. Vincent Poor & Qingsong Wen & Stefan Zohren, 2024. "A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges," Papers 2406.11903, arXiv.org.
    14. Xi Zhang & Yunjia Zhang & Senzhang Wang & Yuntao Yao & Binxing Fang & Philip S. Yu, 2018. "Improving Stock Market Prediction via Heterogeneous Information Fusion," Papers 1801.00588, arXiv.org.
    15. Tim Loughran & Bill Mcdonald, 2011. "When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10‐Ks," Journal of Finance, American Finance Association, vol. 66(1), pages 35-65, February.
    16. Yupeng Cao & Zhi Chen & Prashant Kumar & Qingyun Pei & Yangyang Yu & Haohang Li & Fabrizio Dimino & Lorenzo Ausiello & K. P. Subbalakshmi & Papa Momar Ndiaye, 2024. "RiskLabs: Predicting Financial Risk Using Large Language Model based on Multimodal and Multi-Sources Data," Papers 2404.07452, arXiv.org, revised May 2025.
    17. Minchae Song & Kyung‐shik Shin, 2019. "Forecasting economic indicators using a consumer sentiment index: Survey‐based versus text‐based data," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 38(6), pages 504-518, September.
    18. Eleni Kalamara & Arthur Turrell & Chris Redl & George Kapetanios & Sujit Kapadia, 2022. "Making text count: Economic forecasting using newspaper text," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(5), pages 896-919, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Beckert, Bernd & Kroll, Henning, 2024. "Definition of the research and innovation field of "Artificial Intelligence" and approaches to determining quality," Discussion Papers "Innovation Systems and Policy Analysis" 88, Fraunhofer Institute for Systems and Innovation Research (ISI).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Luca Barbaglia & Sergio Consoli & Sebastiano Manzan, 2024. "Forecasting GDP in Europe with textual data," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 39(2), pages 338-355, March.
    2. Julian Ashwin & Eleni Kalamara & Lorena Saiz, 2024. "Nowcasting Euro area GDP with news sentiment: A tale of two crises," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 39(5), pages 887-905, August.
    3. Martin Baumgaertner & Johannes Zahner, 2021. "Whatever it takes to understand a central banker - Embedding their words using neural networks," MAGKS Papers on Economics 202130, Philipps-Universität Marburg, Faculty of Business Administration and Economics, Department of Economics (Volkswirtschaftliche Abteilung).
    4. Christopher Gerling & Stefan Lessmann, 2023. "Multimodal Document Analytics for Banking Process Automation," Papers 2307.11845, arXiv.org, revised Nov 2023.
    5. Kriebel, Johannes & Stitz, Lennart, 2022. "Credit default prediction from user-generated text in peer-to-peer lending using deep learning," European Journal of Operational Research, Elsevier, vol. 302(1), pages 309-323.
    6. Stolbov, Mikhail & Shchepeleva, Maria & Karminsky, Alexander, 2022. "When central bank research meets Google search: A sentiment index of global financial stress," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 81(C).
    7. Aprigliano, Valentina & Emiliozzi, Simone & Guaitoli, Gabriele & Luciani, Andrea & Marcucci, Juri & Monteforte, Libero, 2023. "The power of text-based indicators in forecasting Italian economic activity," International Journal of Forecasting, Elsevier, vol. 39(2), pages 791-808.
    8. Adämmer, Philipp & Prüser, Jan & Schüssler, Rainer A., 2025. "Forecasting macroeconomic tail risk in real time: Do textual data add value?," International Journal of Forecasting, Elsevier, vol. 41(1), pages 307-320.
    9. Saiz, Lorena & Ashwin, Julian & Kalamara, Eleni, 2021. "Nowcasting euro area GDP with news sentiment: a tale of two crises," Working Paper Series 2616, European Central Bank.
    10. Nyman, Rickard & Kapadia, Sujit & Tuckett, David, 2021. "News and narratives in financial systems: Exploiting big data for systemic risk assessment," Journal of Economic Dynamics and Control, Elsevier, vol. 127(C).
    11. Hanjo Odendaal, 2021. "A machine learning approach to domain specific dictionary generation. An economic time series framework," Working Papers 06/2021, Stellenbosch University, Department of Economics.
    12. Dorinth van Dijk & Jasper de Winter, 2023. "Nowcasting GDP using tone-adjusted time varying news topics: Evidence from the financial press," Working Papers 766, DNB.
    13. Katsafados, Apostolos G. & Leledakis, George N. & Pyrgiotakis, Emmanouil G. & Androutsopoulos, Ion & Fergadiotis, Manos, 2024. "Machine learning in bank merger prediction: A text-based approach," European Journal of Operational Research, Elsevier, vol. 312(2), pages 783-797.
    14. He, Yu & Lu, Shanglin & Wei, Ran & Wang, Shixuan, 2024. "Local media sentiment towards pollution and its effect on corporate green innovation," International Review of Financial Analysis, Elsevier, vol. 94(C).
    15. Limosani, Michele & Millemaci, Emanuele & Mustica, Paolo, 2025. "Do green policies enhance short-term economic growth? Assessing EU Recovery and Resilience Plans through the lens of Sustainable Development Goals," Economic Modelling, Elsevier, vol. 147(C).
    16. Elizaveta Volgina, 2025. "Forecasting Inflation Using News Indices," Russian Journal of Money and Finance, Bank of Russia, vol. 84(1), pages 26-59, March.
    17. Luiz Renato Lima & Lucas Lúcio Godeiro & Mohammed Mohsin, 2021. "Time-Varying Dictionary and the Predictive Power of FED Minutes," Computational Economics, Springer;Society for Computational Economics, vol. 57(1), pages 149-181, January.
    18. María del Pilar Cruz N. & Hugo Peralta V. & Juan Pablo Cova M., 2022. "Utilización de noticias de prensa como indicador de confianza económica en tiempo real," Working Papers Central Bank of Chile 938, Central Bank of Chile.
    19. Maureen Cowhey & Seung Jung Lee & Thomas Popeck Spiller & Cindy M. Vojtech, 2022. "Sentiment in Bank Examination Reports and Bank Outcomes," Finance and Economics Discussion Series 2022-077, Board of Governors of the Federal Reserve System (U.S.).
    20. Abdul Baki, Ghina & Beland, Louis-Philippe & Yazbeck, Myra & Zayat, Aline, 2025. "Reactance, rationalization, and women's rights for safe abortion: Evidence from Roe vs. Wade's overturn," Journal of Economic Behavior & Organization, Elsevier, vol. 233(C).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2407.17624. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.