IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2305.08524.html
   My bibliography  Save this paper

Measuring Consistency in Text-based Financial Forecasting Models

Author

Listed:
  • Linyi Yang
  • Yingpeng Ma
  • Yue Zhang

Abstract

Financial forecasting has been an important and active area of machine learning research, as even the most modest advantage in predictive accuracy can be parlayed into significant financial gains. Recent advances in natural language processing (NLP) bring the opportunity to leverage textual data, such as earnings reports of publicly traded companies, to predict the return rate for an asset. However, when dealing with such a sensitive task, the consistency of models -- their invariance under meaning-preserving alternations in input -- is a crucial property for building user trust. Despite this, current financial forecasting methods do not consider consistency. To address this problem, we propose FinTrust, an evaluation tool that assesses logical consistency in financial text. Using FinTrust, we show that the consistency of state-of-the-art NLP models for financial forecasting is poor. Our analysis of the performance degradation caused by meaning-preserving alternations suggests that current text-based methods are not suitable for robustly predicting market information. All resources are available at https://github.com/yingpengma/fintrust.

Suggested Citation

  • Linyi Yang & Yingpeng Ma & Yue Zhang, 2023. "Measuring Consistency in Text-based Financial Forecasting Models," Papers 2305.08524, arXiv.org, revised Jun 2023.
    • Handle: RePEc:arx:papers:2305.08524
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2305.08524
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Fuli Feng & Huimin Chen & Xiangnan He & Ji Ding & Maosong Sun & Tat-Seng Chua, 2018. "Enhancing Stock Movement Prediction with Adversarial Training," Papers 1810.09936, arXiv.org, revised Jun 2019.
    2. Xi Zhang & Yunjia Zhang & Senzhang Wang & Yuntao Yao & Binxing Fang & Philip S. Yu, 2018. "Improving Stock Market Prediction via Heterogeneous Information Fusion," Papers 1801.00588, arXiv.org.
    3. Linyi Yang & Jiazheng Li & Ruihai Dong & Yue Zhang & Barry Smyth, 2022. "NumHTML: Numeric-Oriented Hierarchical Transformer Model for Multi-task Financial Forecasting," Papers 2201.01770, arXiv.org.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kamaladdin Fataliyev & Aneesh Chivukula & Mukesh Prasad & Wei Liu, 2021. "Stock Market Analysis with Text Data: A Review," Papers 2106.12985, arXiv.org, revised Jul 2021.
    2. Kelvin J. L. Koa & Yunshan Ma & Ritchie Ng & Tat-Seng Chua, 2023. "Diffusion Variational Autoencoder for Tackling Stochasticity in Multi-Step Regression Stock Price Prediction," Papers 2309.00073, arXiv.org, revised Oct 2023.
    3. Yu Zhao & Huaming Du & Ying Liu & Shaopeng Wei & Xingyan Chen & Fuzhen Zhuang & Qing Li & Ji Liu & Gang Kou, 2022. "Stock Movement Prediction Based on Bi-typed Hybrid-relational Market Knowledge Graph via Dual Attention Networks," Papers 2201.04965, arXiv.org, revised Jan 2022.
    4. Zhou, Zhongbao & Gao, Meng & Liu, Qing & Xiao, Helu, 2020. "Forecasting stock price movements with multiple data sources: Evidence from stock market in China," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 542(C).
    5. Hengxu Lin & Dong Zhou & Weiqing Liu & Jiang Bian, 2021. "Learning Multiple Stock Trading Patterns with Temporal Routing Adaptor and Optimal Transport," Papers 2106.12950, arXiv.org, revised Jun 2021.
    6. Jiezhu Cheng & Kaizhu Huang & Zibin Zheng, 2023. "Can Perturbations Help Reduce Investment Risks? Risk-Aware Stock Recommendation via Split Variational Adversarial Training," Papers 2304.11043, arXiv.org, revised Jan 2024.
    7. Ashwini Saini & Anoop Sharma, 2022. "Predicting the Unpredictable: An Application of Machine Learning Algorithms in Indian Stock Market," Annals of Data Science, Springer, vol. 9(4), pages 791-799, August.
    8. Hengxu Lin & Dong Zhou & Weiqing Liu & Jiang Bian, 2021. "Deep Risk Model: A Deep Learning Solution for Mining Latent Risk Factors to Improve Covariance Matrix Estimation," Papers 2107.05201, arXiv.org, revised Oct 2021.
    9. Karolina Sowinska & Pranava Madhyastha, 2020. "A Tweet-based Dataset for Company-Level Stock Return Prediction," Papers 2006.09723, arXiv.org.
    10. Xiao Yang & Weiqing Liu & Dong Zhou & Jiang Bian & Tie-Yan Liu, 2020. "Qlib: An AI-oriented Quantitative Investment Platform," Papers 2009.11189, arXiv.org.
    11. Jiexia Ye & Juanjuan Zhao & Kejiang Ye & Chengzhong Xu, 2020. "Multi-Graph Convolutional Network for Relationship-Driven Stock Movement Prediction," Papers 2005.04955, arXiv.org, revised Oct 2020.
    12. Abbas Haider & Hui Wang & Bryan Scotney & Glenn Hawe, 2022. "Predictive Market Making via Machine Learning," SN Operations Research Forum, Springer, vol. 3(1), pages 1-21, March.
    13. Li-Chen Cheng & Wei-Ting Lu & Benjamin Yeo, 2023. "Predicting abnormal trading behavior from internet rumor propagation: a machine learning approach," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 9(1), pages 1-23, December.
    14. Zhou, Zhongbao & Gao, Meng & Xiao, Helu & Wang, Rui & Liu, Wenbin, 2021. "Big data and portfolio optimization: A novel approach integrating DEA with multiple data sources," Omega, Elsevier, vol. 104(C).
    15. Francisco de Arriba-P'erez & Silvia Garc'ia-M'endez & Jos'e A. Regueiro-Janeiro & Francisco J. Gonz'alez-Casta~no, 2024. "Detection of financial opportunities in micro-blogging data with a stacked classification system," Papers 2404.07224, arXiv.org.
    16. Omer Berat Sezer & Mehmet Ugur Gudelek & Ahmet Murat Ozbayoglu, 2019. "Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019," Papers 1911.13288, arXiv.org.
    17. Wen, Danyan & Ma, Chaoqun & Wang, Gang-Jin & Wang, Senzhang, 2018. "Investigating the features of pairs trading strategy: A network perspective on the Chinese stock market," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 505(C), pages 903-918.
    18. Qianqian Xie & Weiguang Han & Yanzhao Lai & Min Peng & Jimin Huang, 2023. "The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges," Papers 2304.05351, arXiv.org, revised Apr 2023.
    19. Hui Niu & Siyuan Li & Jian Li, 2022. "MetaTrader: An Reinforcement Learning Approach Integrating Diverse Policies for Portfolio Optimization," Papers 2210.01774, arXiv.org.
    20. Shwai He & Shi Gu, 2021. "Multi-modal Attention Network for Stock Movements Prediction," Papers 2112.13593, arXiv.org, revised Oct 2022.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2305.08524. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.