IDEAS home Printed from https://ideas.repec.org/p/fip/fedkrw/97255.html
   My bibliography  Save this paper

Evaluating Local Language Models: An Application to Bank Earnings Calls

Author

Abstract

This study evaluates the performance of local large language models (LLMs) in interpreting financial texts, compared with closed-source, cloud-based models. We first introduce new benchmarking tasks for assessing LLM performance in analyzing financial and economic texts and explore the refinements needed to improve its performance. Our benchmarking results suggest local LLMs are a viable tool for general natural language processing analysis of these texts. We then leverage local LLMs to analyze the tone and substance of bank earnings calls in the post-pandemic era, including calls conducted during the banking stress of early 2023. We analyze remarks in bank earnings calls in terms of topics discussed, overall sentiment, temporal orientation, and vagueness. We find that after the banking stress in early 2023, banks tended to converge to a similar set of topics for discussion and to espouse a distinctly less positive sentiment.

Suggested Citation

  • Thomas R. Cook & Sophia Kazinnik & Anne Lundgaard Hansen & Peter McAdam, 2023. "Evaluating Local Language Models: An Application to Bank Earnings Calls," Research Working Paper RWP 23-12, Federal Reserve Bank of Kansas City.
    • Handle: RePEc:fip:fedkrw:97255
    as

    Download full text from publisher

    File URL: https://www.kansascityfed.org/Research%20Working%20Papers/documents/9862/rwp23-12cookkazinnikhansenmcadam.pdf
    File Function: Full Text
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. In-Koo Cho & David M. Kreps, 1987. "Signaling Games and Stable Equilibria," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 102(2), pages 179-221.
    2. De Amicis, Chiara & Falconieri, Sonia & Tastan, Mesut, 2021. "Sentiment analysis and gender differences in earnings conference calls," Journal of Corporate Finance, Elsevier, vol. 71(C).
    3. Alejandro Lopez-Lira & Yuehua Tang, 2023. "Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models," Papers 2304.07619, arXiv.org, revised Sep 2024.
    4. Chris Chatfield, 1995. "Model Uncertainty, Data Mining and Statistical Inference," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 158(3), pages 419-444, May.
    5. Pekka Malo & Ankur Sinha & Pekka Korhonen & Jyrki Wallenius & Pyry Takala, 2014. "Good debt or bad debt: Detecting semantic orientations in economic texts," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(4), pages 782-796, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yuqi Nie & Yaxuan Kong & Xiaowen Dong & John M. Mulvey & H. Vincent Poor & Qingsong Wen & Stefan Zohren, 2024. "A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges," Papers 2406.11903, arXiv.org.
    2. Baptiste Lefort & Eric Benhamou & Jean-Jacques Ohana & David Saltiel & Beatrice Guez, 2024. "Optimizing Performance: How Compact Models Match or Exceed GPT's Classification Capabilities through Fine-Tuning," Papers 2409.11408, arXiv.org.
    3. Claudia García-García & Catalina B. García-García & Román Salmerón, 2021. "Confronting collinearity in environmental regression models: evidence from world data," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 30(3), pages 895-926, September.
    4. Andrés Perea & Elias Tsakas, 2019. "Limited focus in dynamic games," International Journal of Game Theory, Springer;Game Theory Society, vol. 48(2), pages 571-607, June.
    5. Anders Gustafsson, 2019. "Busy doing nothing: why politicians implement inefficient policies," Constitutional Political Economy, Springer, vol. 30(3), pages 282-299, September.
    6. Dionne, G. & Doherty, N., 1991. "Adverse Selection In Insurance Markets: A Selective Survey," Cahiers de recherche 9105, Centre interuniversitaire de recherche en économie quantitative, CIREQ.
    7. Cao, Melanie & Shi, Shouyong, 2006. "Signaling in the Internet craze of initial public offerings," Journal of Corporate Finance, Elsevier, vol. 12(4), pages 818-833, September.
    8. Mario Gilli & Yuan Li, 2014. "Accountability in One-Party Government: Rethinking the Success of Chinese Economic Reform," Journal of Institutional and Theoretical Economics (JITE), Mohr Siebeck, Tübingen, vol. 170(4), pages 616-645, December.
    9. Thomas de Haan & Theo Offerman & Randolph Sloof, 2015. "Money Talks? An Experimental Investigation Of Cheap Talk And Burned Money," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 56(4), pages 1385-1426, November.
    10. Espínola-Arredondo, Ana & Muñoz-García, Félix, 2013. "When does environmental regulation facilitate entry-deterring practices," Journal of Environmental Economics and Management, Elsevier, vol. 65(1), pages 133-152.
    11. Eduardo Perez & Delphine Prady, 2012. "Complicating to Persuade?," Working Papers hal-03583827, HAL.
    12. Chou, Ping & Chuang, Howard Hao-Chun & Chou, Yen-Chun & Liang, Ting-Peng, 2022. "Predictive analytics for customer repurchase: Interdisciplinary integration of buy till you die modeling and machine learning," European Journal of Operational Research, Elsevier, vol. 296(2), pages 635-651.
    13. Eduardo Perez-Richet, 2014. "Interim Bayesian Persuasion: First Steps," American Economic Review, American Economic Association, vol. 104(5), pages 469-474, May.
    14. V. Bhaskar & Caroline Thomas, 2019. "The Culture of Overconfidence," American Economic Review: Insights, American Economic Association, vol. 1(1), pages 95-110, June.
    15. Fabrizio Adriani & Giancarlo Marini & Pasquale Scaramozzino, 2009. "The Inflationary Consequences of a Currency Changeover on the Catering Sector: Evidence from the Michelin Red Guide," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 71(1), pages 111-133, February.
    16. Vaccari, Federico, 2023. "Competition in costly talk," Journal of Economic Theory, Elsevier, vol. 213(C).
    17. Pim Heijnen, 2013. "Informative advertising by an environmental group," Journal of Economics, Springer, vol. 108(3), pages 249-272, April.
    18. Dutta, Bhaskar & Vohra, Rajiv, 2005. "Incomplete information, credibility and the core," Mathematical Social Sciences, Elsevier, vol. 50(2), pages 148-165, September.
    19. Sai Ding & John Knight, 2011. "Why has China Grown So Fast? The Role of Physical and Human Capital Formation," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 73(2), pages 141-174, April.
    20. Riccardo (Jack) Lucchetti & Luca Pedini, 2020. "ParMA: Parallelised Bayesian Model Averaging for Generalised Linear Models," Working Papers 2020:28, Department of Economics, University of Venice "Ca' Foscari".

    More about this item

    Keywords

    data; large language models; quantitative methods; banking and finance;
    All these keywords.

    JEL classification:

    • C45 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - Neural Networks and Related Topics
    • G21 - Financial Economics - - Financial Institutions and Services - - - Banks; Other Depository Institutions; Micro Finance Institutions; Mortgages

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fip:fedkrw:97255. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Zach Kastens (email available below). General contact details of provider: https://edirc.repec.org/data/frbkcus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.