IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2608.08634.html

Can Open-Weight Models Compete on Financial Text Comprehension?

Author

Listed:
  • Jan Sporer

Abstract

Open-weight language models from Chinese AI labs caught up on benchmarks relative to proprietary frontier models in recent months. Yet their reliability on real-world financial tasks remains largely untested. We updated the Financial Touchstone benchmark, which now has 2,967 question context-answer triplets across 495 international annual reports. We also apply a new set of models on the benchmark, expanding coverage from eleven to twenty models across ten providers, including recent open-weight models such as GLM 4.7, GLM 5, Kimi K2.6, and DeepSeek V3.2, as well as Alibaba's proprietary flagship Qwen3-Max. Anthropic's Claude Opus 4.6 achieves the highest accuracy (88.4%), while Google's Gemini 2.5 Pro maintains the lowest hallucination rate (0.08%). Notably, the open-weight Kimi K2.6 ranks third in accuracy, and the non-reasoning models GLM 5 and Mistral 3 rank fourth and fifth, challenging the assumption that reasoning architectures or proprietary weights are a prerequisite for strong financial comprehension. Information retrieval remains the primary bottleneck, accounting for 48.9% of all failures. We also document a new finding: geopolitical content filters in Chinese models refuse legitimate financial questions (0.08% of attempts), sometimes without clear reason, and the refusal behavior depends on the access route as much as on the model. The complete dataset and evaluation framework are publicly available.

Suggested Citation

  • Jan Sporer, 2026. "Can Open-Weight Models Compete on Financial Text Comprehension?," Papers 2608.08634, arXiv.org.
  • Handle: RePEc:arx:papers:2608.08634
    as

    Download full text from publisher

    File URL: https://arxiv.org/pdf/2608.08634
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Cristi A. Gleason & W. Bruce Johnson & Haidan Li, 2013. "Valuation Model Use and the Price Target Performance of Sell†Side Equity Analysts," Contemporary Accounting Research, John Wiley & Sons, vol. 30(1), pages 80-115, March.
    2. Stefano Bonini & Laura Zanetti & Roberto Bianchini & Antonio Salvi, 2010. "Target Price Accuracy in Equity Research," Journal of Business Finance & Accounting, Wiley Blackwell, vol. 37(9‐10), pages 1177-1217, November.
    3. Womack, Kent L, 1996. "Do Brokerage Analysts' Recommendations Have Investment Value?," Journal of Finance, American Finance Association, vol. 51(1), pages 137-167, March.
    4. Asquith, Paul & Mikhail, Michael B. & Au, Andrea S., 2005. "Information content of equity analyst reports," Journal of Financial Economics, Elsevier, vol. 75(2), pages 245-282, February.
    5. Brad Barber & Reuven Lehavy & Maureen McNichols & Brett Trueman, 2001. "Can Investors Profit from the Prophets? Security Analyst Recommendations and Stock Returns," Journal of Finance, American Finance Association, vol. 56(2), pages 531-563, April.
    6. Tim Loughran & Bill Mcdonald, 2011. "When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10‐Ks," Journal of Finance, American Finance Association, vol. 66(1), pages 35-65, February.
    7. Wayne R. Landsman & Edward L. Maydew, 2002. "Has the Information Content of Quarterly Earnings Announcements Declined in the Past Three Decades?," Journal of Accounting Research, John Wiley & Sons, Ltd., vol. 40(3), pages 797-808, June.
    8. Pekka Malo & Ankur Sinha & Pekka Korhonen & Jyrki Wallenius & Pyry Takala, 2014. "Good debt or bad debt: Detecting semantic orientations in economic texts," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(4), pages 782-796, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Adria Pop & Jan Sporer, 2024. "The Structure of Financial Equity Research Reports -- Identification of the Most Frequently Asked Questions in Financial Analyst Reports to Automate Equity Research Using Llama 3 and GPT-4," Papers 2407.18327, arXiv.org, revised Jun 2025.
    2. Samie Ahmed Sayed, 2016. "Are Stars Worth Following? Measuring the Target Price Predictive Ability of Star Analysts in an Emerging Market," IIM Kozhikode Society & Management Review, , vol. 5(2), pages 173-185, July.
    3. Tuan Ho & Ruby Brownen‐Trinh & Fangming Xu, 2021. "The information content of target price forecasts: Evidence from mergers and acquisitions," Journal of Business Finance & Accounting, Wiley Blackwell, vol. 48(5-6), pages 1134-1171, May.
    4. Suresh Kadam & Madhvi Sethi, 2026. "Mapping the Landscape of Analyst Stock Recommendations: A Bibliometric Analysis," Journal of the Knowledge Economy, Springer;Portland International Center for Management of Engineering and Technology (PICMET), vol. 17(2), pages 3740-3793, April.
    5. Hiren Patel, 2021. "Target Price Achievement and Target Price Accuracy Models: An Analysis of Advisory Firms’ Recommendation for the Indian Banking Stocks," Global Business Review, International Management Institute, vol. 22(2), pages 459-473, April.
    6. Christoph Lohrmann & Alena Lohrmann, 2021. "Accuracy and Predictive Power of Sell-Side Target Prices for Global Clean Energy Companies," Sustainability, MDPI, vol. 13(22), pages 1-27, November.
    7. Imam, Shahed & Chan, Jacky & Shah, Syed Zulfiqar Ali, 2013. "Equity valuation models and target price accuracy in Europe: Evidence from equity reports," International Review of Financial Analysis, Elsevier, vol. 28(C), pages 9-19.
    8. repec:cgr:cgsser:03-03 is not listed on IDEAS
    9. Leonardo Fernandez, 2012. "Price Discovery, Investor Distraction and Analyst Recommendations Under Continuous Disclosure Requirements in Australia," PhD Thesis, Finance Discipline Group, UTS Business School, University of Technology, Sydney, number 1-2012, January-A.
    10. Devos, Erik & Hao, Wei & Prevost, Andrew K. & Wongchoti, Udomsak, 2015. "Stock return synchronicity and the market response to analyst recommendation revisions," Journal of Banking & Finance, Elsevier, vol. 58(C), pages 376-389.
    11. Osman Yukselturk & Jon Tucker, 2015. "The impact of analyst sentiment on UK stock recommendations and target prices," Accounting and Business Research, Taylor & Francis Journals, vol. 45(6-7), pages 869-904, December.
    12. Caylor, Marcus & Cecchini, Mark & Winchel, Jennifer, 2017. "Analysts' qualitative statements and the profitability of favorable investment recommendations," Accounting, Organizations and Society, Elsevier, vol. 57(C), pages 33-51.
    13. Leonardo Fernandez, 2012. "Price Discovery, Investor Distraction and Analyst Recommendations Under Continuous Disclosure Requirements in Australia," PhD Thesis, Finance Discipline Group, UTS Business School, University of Technology, Sydney, number 3, July-Dece.
    14. Stefan Kanne & Jan Klobucnik & Daniel Kreutzmann & Soenke Sievers, 2012. "To buy or not to buy? The value of contradictory analyst signals," Financial Markets and Portfolio Management, Springer;Swiss Society for Financial Market Research, vol. 26(4), pages 405-428, December.
    15. Gu, Chen & Guo, Xu & Zhang, Chengping, 2022. "Analyst target price revisions and institutional herding," International Review of Financial Analysis, Elsevier, vol. 82(C).
    16. Ekene S. Aguegboh & Uchenna C. Onuoha & Poojan Patel, 2026. "Assessing the relevance of sell‐side analyst recommendations," Review of Financial Economics, John Wiley & Sons, vol. 44(1), January.
    17. Hess, Dieter & Kreutzmann, Daniel & Pucker, Oliver, 2011. "Projected earnings accuracy and the profitability of stock recommendations," CFR Working Papers 10-17 [rev.], University of Cologne, Centre for Financial Research (CFR).
    18. Hsieh, Wen-liang Gideon & Lee, Chin-Shen, 2021. "Who reacts to what information in securities analyst reports? Direct evidence from the investor trade imbalance," Pacific-Basin Finance Journal, Elsevier, vol. 65(C).
    19. Engelberg, Joseph & McLean, R. David & Pontiff, Jeffrey, 2020. "Analysts and anomalies," Journal of Accounting and Economics, Elsevier, vol. 69(1).
    20. Peter F. Pope & Tong Wang, 2023. "Analyst ability and research effort: non-EPS forecast provision as a research quality signal," Review of Accounting Studies, Springer, vol. 28(3), pages 1263-1315, September.
    21. Thabang Mokoaleli-Mokoteli & Richard J. Taffler & Vineet Agarwal, 2009. "Behavioural Bias and Conflicts of Interest in Analyst Stock Recommendations," Journal of Business Finance & Accounting, Wiley Blackwell, vol. 36(3-4), pages 384-418.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2608.08634. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: https://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.