IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2606.08285.html

Execution Realism and Reproducibility in LLM-Based Trading Systems: A Systematic Scoping Review and Evidence Audit

Author

Listed:
  • Junyi Yao
  • Zihao Zheng
  • Baichuan Li
  • Jiayu Long

Abstract

Execution realism remains weakly standardized in research on large-language-model-based trading systems, limiting comparison and reproduction across backtests, simulations, and portfolio benchmarks. This article presents a PRISMA-ScR systematic scoping review with a nested execution-reproducibility evidence audit. Eight reproducible query families returned 796 query-level records; DOI/title deduplication left 687 unique records, 101 reports were sought for retrieval, and 59 full texts were recovered through direct and fallback open-access routes. Fifty-three reports are provisionally eligible for expanded evidence charting, subject to reconciliation of two blind author-validation packets. The charting framework separately evaluates point-in-time controls, temporal splits, held-out evaluation, trading costs and turnover, execution semantics, universe construction, architecture reporting, and artifact availability. Direct-trading, portfolio or alpha-construction, and benchmark studies are retained as distinct descriptive subgroups rather than pooled as a common performance estimand. Search responses, retrieval attempts, file hashes, evidence excerpts, screening decisions, and coding worksheets are archived with the review materials. Final field-level counts are intentionally withheld until author validation is complete.

Suggested Citation

  • Junyi Yao & Zihao Zheng & Baichuan Li & Jiayu Long, 2026. "Execution Realism and Reproducibility in LLM-Based Trading Systems: A Systematic Scoping Review and Evidence Audit," Papers 2606.08285, arXiv.org, revised Oct 2026.
  • Handle: RePEc:arx:papers:2606.08285
    as

    Download full text from publisher

    File URL: https://arxiv.org/pdf/2606.08285
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Haohang Li & Yupeng Cao & Yangyang Yu & Shashidhar Reddy Javaji & Zhiyang Deng & Yueru He & Yuechen Jiang & Zining Zhu & Koduvayur Subbalakshmi & Guojun Xiong & Jimin Huang & Lingfei Qian & Xueqing Pe, 2024. "INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent," Papers 2412.18174, arXiv.org.
    2. Terence Lim & Kumar Muthuraman & Michael Sury, 2026. "QRAFTI: An Agentic Framework for Empirical Research in Quantitative Finance," Papers 2604.18500, arXiv.org.
    3. Thanos Konstantinidis & Giorgos Iacovides & Mingxue Xu & Tony G. Constantinides & Danilo Mandic, 2024. "FinLlama: Financial Sentiment Classification for Algorithmic Trading Applications," Papers 2403.12285, arXiv.org.
    4. Alejandro Lopez-Lira, 2025. "Can Large Language Models Trade? Testing Financial Theories with LLM Agents in Market Simulations," Papers 2504.10789, arXiv.org.
    5. Dat Mai, 2024. "StockGPT: A GenAI Model for Stock Prediction and Trading," Papers 2404.05101, arXiv.org, revised Oct 2024.
    6. Saizhuo Wang & Hang Yuan & Lionel M. Ni & Jian Guo, 2024. "QuantAgent: Seeking Holy Grail in Trading by Self-Improving Large Language Model," Papers 2402.03755, arXiv.org.
    7. Chong Zhang & Xinyi Liu & Zhongmou Zhang & Mingyu Jin & Lingyao Li & Zhenting Wang & Wenyue Hua & Dong Shu & Suiyuan Zhu & Xiaobo Jin & Sujian Li & Mengnan Du & Yongfeng Zhang, 2024. "When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments," Papers 2407.18957, arXiv.org, revised Jun 2026.
    8. Yuxuan Zhao & Sijia Chen & Ningxin Su, 2026. "PortBench: A Correlation-Aware, Full-Pipeline Benchmark for LLM-Driven Portfolio Management," Papers 2605.27887, arXiv.org, revised Oct 2026.
    9. Jean Lee & Nicholas Stevens & Soyeon Caren Han & Minseok Song, 2024. "A Survey of Large Language Models in Finance (FinLLMs)," Papers 2402.02315, arXiv.org.
    10. Adam Darmanin & Vince Vella, 2025. "Language Model Guided Reinforcement Learning in Quantitative Trading," Papers 2508.02366, arXiv.org, revised Oct 2025.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Xavier Fonseca, 2026. "Look-Ahead-Freedom as Temporal Non-Interference: A Verifiable Correctness Property for Backtesting and Agentic Trading Pipelines," Papers 2607.04958, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Han Ding & Yinheng Li & Junhao Wang & Hang Chen & Doudou Guo & Yunbai Zhang, 2024. "Large Language Model Agent in Financial Trading: A Survey," Papers 2408.06361, arXiv.org, revised Mar 2026.
    2. Zhi Yang & Lingfeng Zeng & Fangqi Lou & Qi Qi & Wei Zhang & Zhenyu Wu & Zhenxiong Yu & Jun Han & Zhiheng Jin & Lejie Zhang & Xiaoming Huang & Xiaolong Liang & Zheng Wei & Junbo Zou & Dongpo Cheng & Zh, 2026. "UniFinEval: Towards Unified Evaluation of Financial Multimodal Models across Text, Images and Videos," Papers 2601.22162, arXiv.org.
    3. Mostapha Benhenda, 2026. "Look-Ahead-Bench: a Standardized Benchmark of Look-ahead Bias in Point-in-Time LLMs for Finance," Papers 2601.13770, arXiv.org.
    4. Benjamin Coriat & Eric Benhamou, 2025. "HARLF: Hierarchical Reinforcement Learning and Lightweight LLM-Driven Sentiment Integration for Financial Portfolio Optimization," Papers 2507.18560, arXiv.org.
    5. Liyuan Chen & Shuoling Liu & Jiangpeng Yan & Xiaoyu Wang & Henglin Liu & Chuang Li & Kecheng Jiao & Jixuan Ying & Yang Veronica Liu & Qiang Yang & Xiu Li, 2025. "Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges," Papers 2507.18577, arXiv.org, revised Dec 2025.
    6. Tirulo, Aschalew & Yadav, Monika & Lolamo, Mathewos & Chauhan, Siddhartha & Siano, Pierluigi & Shafie-khah, Miadreza, 2026. "Beyond automation: Unveiling the potential of agentic intelligence," Renewable and Sustainable Energy Reviews, Elsevier, vol. 226(PA).
    7. Rubén Fernández-Fuertes, 2025. "Monetary Policy Shocks: A New Hope. Large Language Models and Central Bank Communication," BAFFI CAREFIN Working Papers 25257, BAFFI CAREFIN, Centre for Applied Research on International Markets Banking Finance and Regulation, Universita' Bocconi, Milano, Italy.
    8. Kunjesh Parekh & Anil Kumar Tiwari & Divya Saxena, 2026. "CIFQA: A Deterministic Tool-Grounded Multi-Agent LLM Framework for Financial Query Answering," Papers 2608.26114, arXiv.org.
    9. Hamidou Tembine & Manzoor Ahmed Khan & Issa Bamia, 2024. "Mean-Field-Type Transformers," Mathematics, MDPI, vol. 12(22), pages 1-51, November.
    10. Matthew Francis Dixon, 2026. "Model Validation of Agentic AI Systems: A POMDP-Based Framework for Belief-State, Forecast, and Policy Validation," Papers 2606.17383, arXiv.org.
    11. Zichen Chen & Jiaao Chen & Jianda Chen & Misha Sra, 2025. "Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Risk," Papers 2502.15865, arXiv.org, revised Jun 2025.
    12. Kassiani Papasotiriou & Srijan Sood & Shayleen Reynolds & Tucker Balch, 2024. "AI in Investment Analysis: LLMs for Equity Stock Ratings," Papers 2411.00856, arXiv.org.
    13. Zhenyu Gao & Wenxi Jiang & Yutong Yan, 2026. "Debiasing LLMs by Fine-tuning," Papers 2604.02921, arXiv.org, revised May 2026.
    14. Junhua Liu, 2024. "A Survey of Financial AI: Architectures, Advances and Open Challenges," Papers 2411.12747, arXiv.org.
    15. Filippo Gusella & Eugenio Vicario, 2025. "Generative Agents and Expectations: Do LLMs Align with Heterogeneous Agent Models?," Working Papers - Economics wp2025_18.rdf, Universita' degli Studi di Firenze, Dipartimento di Scienze per l'Economia e l'Impresa.
    16. Aadi Singhi, 2025. "An Adaptive Multi Agent Bitcoin Trading System," Papers 2510.08068, arXiv.org, revised Nov 2025.
    17. Yingjian Pan & Xiaowei Ding & Kay Giesecke, 2026. "Agentic Empirical Asset Pricing: Methodological Foundations," Papers 2609.00731, arXiv.org.
    18. Wang, Lu & Yan, Junbo & Yan, Jiang & Liu, Yan, 2026. "Towards intelligent safety assessment under blast loading: A large language model-powered multi-agent framework," Reliability Engineering and System Safety, Elsevier, vol. 272(P2).
    19. Simeon Allmendinger & Lukas Bonenberger & Kathrin Endres & Dominik Fetzer & Henner Gimpel & Niklas Kühl, 2026. "Multi-agent AI," Electronic Markets, Springer;IIM University of St. Gallen, vol. 36(1), pages 1-18, December.
    20. Ryuji Hashimoto & Takehiro Takayanagi & Masahiro Suzuki & Kiyoshi Izumi, 2026. "LLM agents reveal how human bias shapes path-dependent market dynamics," Journal of Computational Social Science, Springer, vol. 9(2), pages 1-26, May.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2606.08285. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: https://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.