IDEAS home Printed from https://ideas.repec.org/p/bhd/dpaper/202501.html

Let U​​s Ask a Gen AI Chatbot: Machine Versus Bank Survey Results

Author

Listed:
  • Veronica B. Bayangos

    (Bangko Sentral ng Pilipinas)

  • Neil Fidelle G. Lomibao

    (Bangko Sentral ng Pilipinas)

  • Ralph Joshua P. Macarasig

    (Bangko Sentral ng Pilipinas)

Abstract

This study evaluates the performance of two generative pre-trained transformer (GPT) models, ChatGPT 4o and Gemini Pro 1.5, in responding to a banking sector outlook survey designed to support supervisory policy assessment. The study is structured around three main exercises. The first classifies questions related to the analysis of financial risks, challenges, and outlooks. The second employs prompt engineering to mitigate biases and address potential hallucinations in the models. The third constructs Cosine Similarity Indices (CSIs) to compare the responses of the GPT models with those of banks. The findings suggest that both models can serve as valuable tools for assessing a bank’s financial performance, risks, and macroeconomic outlook. These results underscore their potential to complement a supervisor’s analytical tasks. However, the observed differences in responses may point to the complex and potentially non-linear relationships that underlie bank performance, thereby reinforcing the continued importance of human judgment in bank supervision.

Suggested Citation

  • Veronica B. Bayangos & Neil Fidelle G. Lomibao & Ralph Joshua P. Macarasig, 2025. "Let U​​s Ask a Gen AI Chatbot: Machine Versus Bank Survey Results," BSP Discussion Paper Series 202501, Bangko Sentral ng Pilipinas.
    • Handle: RePEc:bhd:dpaper:202501
    as

    Download full text from publisher

    File URL: https://www.bsp.gov.ph/Sites/researchsite/Publications/BSP-Discussion-Papers/DP202501.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Bertsch, Christoph & Hull, Isaiah & Lumsdaine, Robin L. & Zhang, Xin, 2025. "Central bank mandates and monetary policy stances: Through the lens of Federal Reserve speeches," Journal of Econometrics, Elsevier, vol. 249(PC).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alexandra Piller & Marc Schranz & Larissa Schwaller, 2025. "Using Natural Language Processing to Identify Monetary Policy Shocks," Working Papers 25.05, Swiss National Bank, Study Center Gerzensee.
    2. Granziera, Eleonora & Larsen, Vegard H. & Meggiorini, Greta & Melosi, Leonardo, 2025. "Speaking of Inflation: The Influence of Fed Speeches on Expectations," CEPR Discussion Papers 20038, Centre for Economic Policy Research.
    3. Chavleishvili, Sulkhan & Engle, Robert F. & Fahr, Stephan & Kremer, Manfred & Lund-Thomsen, Frederik & Manganelli, Simone & Schwaab, Bernd, 2026. "Macro-prudential policy under asymmetric risks: A Bayesian structural quantile VAR approach," Journal of Econometrics, Elsevier, vol. 256(PB).
    4. Anne Lundgaard Hansen, 2026. "Validating Large Language Model Annotations," Finance and Economics Discussion Series 2026-020, Board of Governors of the Federal Reserve System (U.S.).
    5. Tsang, Kwok Ping & Yang, Zichao, 2025. "Agree to disagree: Measuring hidden dissent in FOMC meetings," Journal of Economic Dynamics and Control, Elsevier, vol. 180(C).
    6. Ehrmann, Michael & Tietz, Robin & Visser, Bauke, 2026. "Voting right rotation, speeches, and financial market reactions: Evidence from the U.S. federal open market committee," Journal of International Money and Finance, Elsevier, vol. 165(C).
    7. Leek, Lauren Caroline & Bischl, Simeon, 2024. "How Central Bank Independence Shapes Monetary Policy Communication: A Large Language Model Application," SocArXiv yrhka, Center for Open Science.
    8. Leek, Lauren & Bischl, Simeon, 2025. "How central bank independence shapes monetary policy communication: A Large Language Model application," European Journal of Political Economy, Elsevier, vol. 87(C).
    9. Marcus Buckmann & Ed Hill, 2025. "Improving text classification: logistic regression makes small LLMs strong and explainable ‘tens-of-shot’ classifiers," Bank of England Staff Working Paper series 1127, Bank of England.
    10. repec:osf:socarx:yrhka_v1 is not listed on IDEAS
    11. Kanelis, Dimitrios & Kranzmann, Lars H. & Siklos, Pierre L., 2025. "The financial instability - Monetary policy nexus: Evidence from the FOMC minutes," Discussion Papers 13/2025, Deutsche Bundesbank.
    12. Ille, Sebastian & Nguyen, Dat Thanh & Sanchez Carrera, Edgar J., 2025. "On the stressful interaction of monetary and fiscal targets," International Review of Economics & Finance, Elsevier, vol. 104(C).
    13. Almgren, Pelle, 2026. "Text-Based Identification of Monetary Policy Shocks," Working Papers 2026:10, Lund University, Department of Economics.

    More about this item

    JEL classification:

    • C55 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Large Data Sets: Modeling and Analysis
    • E42 - Macroeconomics and Monetary Economics - - Money and Interest Rates - - - Monetary Systems; Standards; Regimes; Government and the Monetary System
    • E5 - Macroeconomics and Monetary Economics - - Monetary Policy, Central Banking, and the Supply of Money and Credit
    • E61 - Macroeconomics and Monetary Economics - - Macroeconomic Policy, Macroeconomic Aspects of Public Finance, and General Outlook - - - Policy Objectives; Policy Designs and Consistency; Policy Coordination
    • G28 - Financial Economics - - Financial Institutions and Services - - - Government Policy and Regulation

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bhd:dpaper:202501. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MB Pagalunan (email available below). General contact details of provider: https://edirc.repec.org/data/bspgvph.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.