IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2604.25224.html

ValueBlindBench: Agreement-Gated Stress Testing of LLM-Judged Investment Rationales Before Returns Are Observable

Author

Listed:
  • Sidi Chang
  • Peiying Zhu
  • Yuxiao Chen

Abstract

LLM-based financial agents increasingly produce investment rationales before the outcomes needed to evaluate them are observable. This creates a delayed-ground-truth evaluation problem: realized returns remain the eventual arbiter of investment quality, but they arrive too late and are too noisy to guide many model-development and governance decisions. LLM judges offer a tempting shortcut for pre-deployment evaluation of AI-finance systems, but unvalidated judges may reward verbosity, confidence, or rubric mimicry rather than financial judgment. This paper introduces ValueBlindBench, a preregistered agreement-gated stress-test protocol for deciding when LLM-judged investment-rationale claims are publishable, qualified, or invalid. In a controlled market-state capital-allocation prototype with 1,000 honest decision cycles and 100 preregistered adversarial controls (1,100 trajectories, 5,500 judge calls), ValueBlindBench clears the aggregate agreement gate at \(\bar{\kappa}_w = 0.7168\) but prevents several overclaims. Lower-rank systems collapse into a tie-class, one rubric dimension fails the per-dimension gate (\texttt{constraint\_awareness}, \(\bar{\kappa}_w = 0.2022\)), single-judge rankings are family-dependent, and terse-correct rationales receive a \(\Delta = -2.81\) rubric-point penalty relative to honest rationales. A targeted anchor-specificity probe further shows that financial constructs such as constraint awareness are operationally load-bearing. The scientific object is therefore not a leaderboard and not a claim to measure true investment skill. ValueBlindBench is a pre-calibration metrology layer for AI-finance evaluation: it governs whether a proposed LLM-judge-based investment-rationale claim is stable enough, agreed enough, and uncontaminated enough to be reported at all.

Suggested Citation

  • Sidi Chang & Peiying Zhu & Yuxiao Chen, 2026. "ValueBlindBench: Agreement-Gated Stress Testing of LLM-Judged Investment Rationales Before Returns Are Observable," Papers 2604.25224, arXiv.org, revised May 2026.
  • Handle: RePEc:arx:papers:2604.25224
    as

    Download full text from publisher

    File URL: https://arxiv.org/pdf/2604.25224
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Andrew W. Lo, 2002. "The Statistics of Sharpe Ratios," Financial Analysts Journal, Taylor & Francis Journals, vol. 58(4), pages 36-52, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nathan Lassance & Alberto Martín-Utrera & Majeed Simaan, 2024. "The Risk of Expected Utility Under Parameter Uncertainty," Management Science, INFORMS, vol. 70(11), pages 7644-7663, November.
    2. Kamesh Korangi & Christophe Mues & Cristi'an Bravo, 2024. "Large-scale Time-Varying Portfolio Optimisation using Graph Attention Networks," Papers 2407.15532, arXiv.org, revised Feb 2025.
    3. Michele Costola & Bertrand Maillet & Zhining Yuan & Xiang Zhang, 2024. "Mean–variance efficient large portfolios: a simple machine learning heuristic technique based on the two-fund separation theorem," Annals of Operations Research, Springer, vol. 334(1), pages 133-155, March.
    4. Zhao, Shengli, 2024. "Objective acceptability indexes," International Review of Financial Analysis, Elsevier, vol. 95(PC).
    5. Andrew Paskaramoorthy & Terence van Zyl & Tim Gebbie, 2025. "The bias of IID resampled backtests for rolling-window mean-variance portfolios," Papers 2505.06383, arXiv.org.
    6. Fan Zhang & Zhen Li & Sijia Peng & Yu Chen, 2026. "When Alpha Disappears: A One-Switch Benchmark for Decision-Time Leakage in Financial Backtests," Papers 2605.23959, arXiv.org.
    7. Jian'an Zhang, 2025. "FR-LUX: Friction-Aware, Regime-Conditioned Policy Optimization for Implementable Portfolio Management," Papers 2510.02986, arXiv.org.
    8. Doeswijk, Ronald & Swinkels, Laurens, 2026. "The risk and reward of investing," Journal of International Money and Finance, Elsevier, vol. 160(C).
    9. Yufei Sun, 2025. "Does Pair Trading Still Work During Extreme Events? A Comprehensive Empirical Evidence from Chinese Stock Market," Working Papers 2025-23, Faculty of Economic Sciences, University of Warsaw.
    10. Horn, Matthias & Oehler, Andreas, 2024. "Constructing stock portfolios by sorting on ESG ratings: Does the rating provider matter?," International Review of Financial Analysis, Elsevier, vol. 96(PA).
    11. Keith Cuthbertson & Dirk Nitzsche & Niall O'Sullivan, 2010. "Mutual Fund Performance: Measurement and Evidence," Financial Markets, Institutions & Instruments, John Wiley & Sons, vol. 19(2), pages 95-187, May.
    12. Lotfi, Somayyeh & Pagliardi, Giovanni & Paparoditis, Efstathios & Zenios, Stavros A., 2025. "Hedging political risk in international portfolios," European Journal of Operational Research, Elsevier, vol. 322(2), pages 629-646.
    13. Monia Magnani & Massimo Guidolin & Ian Berk, 2024. "Strong vs. stable: the impact of ESG ratings momentum and their volatility on the cost of equity capital," Journal of Asset Management, Palgrave Macmillan, vol. 25(7), pages 666-699, December.
    14. Damir Tokic & Dave Jackson & Karlo Tokic, 2025. "The factor/style index investing: Midcap growth has been the winner in 1995–2024," Journal of Asset Management, Palgrave Macmillan, vol. 26(5), pages 514-522, September.
    15. Alexander Sheppert, 2026. "The GT-Score: A Robust Objective Function for Reducing Overfitting in Data-Driven Trading Strategies," Papers 2602.00080, arXiv.org.
    16. Michael Ashby, 2024. "Are Regulatory Short Sale Data a Profitable Predictor of UK Stock Returns?," JRFM, MDPI, vol. 17(8), pages 1-33, July.
    17. Fieberg, Christian & Liedtke, Gerrit & Zaremba, Adam & Cakici, Nusret, 2025. "A factor model for the cross-section of country equity risk premia," Journal of Banking & Finance, Elsevier, vol. 171(C).
    18. Karl Demers‐Bélanger & Van Son Lai, 2020. "Diversification benefits of cat bonds: An in‐depth examination," Financial Markets, Institutions & Instruments, John Wiley & Sons, vol. 29(5), pages 165-228, December.
    19. Beckmeyer, Heiner & Wiedemann, Timo, 2025. "All Days Are Not Created Equal: Understanding Momentum by Learning to Weight Past Returns," Journal of Banking & Finance, Elsevier, vol. 181(C).
    20. Ian Berk & Massimo Guidolin & Monia Magnani, 2023. "New ESG rating drivers in the cross‐section of European stock returns," Journal of Financial Research, Southern Finance Association;Southwestern Finance Association, vol. 46(S1), pages 133-162, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2604.25224. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: https://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.