IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2305.08524.html
   My bibliography  Save this paper

Measuring Consistency in Text-based Financial Forecasting Models

Author

Listed:
  • Linyi Yang
  • Yingpeng Ma
  • Yue Zhang

Abstract

Financial forecasting has been an important and active area of machine learning research, as even the most modest advantage in predictive accuracy can be parlayed into significant financial gains. Recent advances in natural language processing (NLP) bring the opportunity to leverage textual data, such as earnings reports of publicly traded companies, to predict the return rate for an asset. However, when dealing with such a sensitive task, the consistency of models -- their invariance under meaning-preserving alternations in input -- is a crucial property for building user trust. Despite this, current financial forecasting methods do not consider consistency. To address this problem, we propose FinTrust, an evaluation tool that assesses logical consistency in financial text. Using FinTrust, we show that the consistency of state-of-the-art NLP models for financial forecasting is poor. Our analysis of the performance degradation caused by meaning-preserving alternations suggests that current text-based methods are not suitable for robustly predicting market information. All resources are available at https://github.com/yingpengma/fintrust.

Suggested Citation

  • Linyi Yang & Yingpeng Ma & Yue Zhang, 2023. "Measuring Consistency in Text-based Financial Forecasting Models," Papers 2305.08524, arXiv.org, revised Jun 2023.
    • Handle: RePEc:arx:papers:2305.08524
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2305.08524
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Xi Zhang & Yunjia Zhang & Senzhang Wang & Yuntao Yao & Binxing Fang & Philip S. Yu, 2018. "Improving Stock Market Prediction via Heterogeneous Information Fusion," Papers 1801.00588, arXiv.org.
    2. Fuli Feng & Huimin Chen & Xiangnan He & Ji Ding & Maosong Sun & Tat-Seng Chua, 2018. "Enhancing Stock Movement Prediction with Adversarial Training," Papers 1810.09936, arXiv.org, revised Jun 2019.
    3. Linyi Yang & Jiazheng Li & Ruihai Dong & Yue Zhang & Barry Smyth, 2022. "NumHTML: Numeric-Oriented Hierarchical Transformer Model for Multi-task Financial Forecasting," Papers 2201.01770, arXiv.org.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kamaladdin Fataliyev & Aneesh Chivukula & Mukesh Prasad & Wei Liu, 2021. "Stock Market Analysis with Text Data: A Review," Papers 2106.12985, arXiv.org, revised Jul 2021.
    2. Kelvin J. L. Koa & Yunshan Ma & Ritchie Ng & Tat-Seng Chua, 2023. "Diffusion Variational Autoencoder for Tackling Stochasticity in Multi-Step Regression Stock Price Prediction," Papers 2309.00073, arXiv.org, revised Oct 2023.
    3. Yu Zhao & Huaming Du & Ying Liu & Shaopeng Wei & Xingyan Chen & Fuzhen Zhuang & Qing Li & Ji Liu & Gang Kou, 2022. "Stock Movement Prediction Based on Bi-typed Hybrid-relational Market Knowledge Graph via Dual Attention Networks," Papers 2201.04965, arXiv.org, revised Jan 2022.
    4. Zhou, Zhongbao & Gao, Meng & Liu, Qing & Xiao, Helu, 2020. "Forecasting stock price movements with multiple data sources: Evidence from stock market in China," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 542(C).
    5. Ashwini Saini & Anoop Sharma, 2022. "Predicting the Unpredictable: An Application of Machine Learning Algorithms in Indian Stock Market," Annals of Data Science, Springer, vol. 9(4), pages 791-799, August.
    6. Xiaolu Wei & Hongbing Ouyang & Muyan Liu, 2022. "Stock index trend prediction based on TabNet feature selection and long short-term memory," PLOS ONE, Public Library of Science, vol. 17(12), pages 1-18, December.
    7. Hengxu Lin & Dong Zhou & Weiqing Liu & Jiang Bian, 2021. "Deep Risk Model: A Deep Learning Solution for Mining Latent Risk Factors to Improve Covariance Matrix Estimation," Papers 2107.05201, arXiv.org, revised Oct 2021.
    8. Karolina Sowinska & Pranava Madhyastha, 2020. "A Tweet-based Dataset for Company-Level Stock Return Prediction," Papers 2006.09723, arXiv.org.
    9. Abbas Haider & Hui Wang & Bryan Scotney & Glenn Hawe, 2022. "Predictive Market Making via Machine Learning," SN Operations Research Forum, Springer, vol. 3(1), pages 1-21, March.
    10. Li-Chen Cheng & Wei-Ting Lu & Benjamin Yeo, 2023. "Predicting abnormal trading behavior from internet rumor propagation: a machine learning approach," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 9(1), pages 1-23, December.
    11. Yihang Fu & Mingyu Zhou & Luyao Zhang, 2024. "DAM: A Universal Dual Attention Mechanism for Multimodal Timeseries Cryptocurrency Trend Forecasting," Papers 2405.00522, arXiv.org.
    12. Francisco de Arriba-P'erez & Silvia Garc'ia-M'endez & Jos'e A. Regueiro-Janeiro & Francisco J. Gonz'alez-Casta~no, 2024. "Detection of financial opportunities in micro-blogging data with a stacked classification system," Papers 2404.07224, arXiv.org.
    13. Omer Berat Sezer & Mehmet Ugur Gudelek & Ahmet Murat Ozbayoglu, 2019. "Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019," Papers 1911.13288, arXiv.org.
    14. Wen, Danyan & Ma, Chaoqun & Wang, Gang-Jin & Wang, Senzhang, 2018. "Investigating the features of pairs trading strategy: A network perspective on the Chinese stock market," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 505(C), pages 903-918.
    15. Shwai He & Shi Gu, 2021. "Multi-modal Attention Network for Stock Movements Prediction," Papers 2112.13593, arXiv.org, revised Oct 2022.
    16. Ramit Sawhney & Shivam Agarwal & Vivek Mittal & Paolo Rosso & Vikram Nanda & Sudheer Chava, 2022. "Cryptocurrency Bubble Detection: A New Stock Market Dataset, Financial Task & Hyperbolic Models," Papers 2206.06320, arXiv.org.
    17. Rosdyana Mangir Irawan Kusuma & Trang-Thi Ho & Wei-Chun Kao & Yu-Yen Ou & Kai-Lung Hua, 2019. "Using Deep Learning Neural Networks and Candlestick Chart Representation to Predict Stock Market," Papers 1903.12258, arXiv.org.
    18. Junran Wu & Ke Xu & Jichang Zhao, 2019. "Online reviews can predict long-term returns of individual stocks," Papers 1905.03189, arXiv.org.
    19. Amit Milstein & Haoran Deng & Guy Revach & Hai Morgenstern & Nir Shlezinger, 2022. "Neural Augmented Kalman Filtering with Bollinger Bands for Pairs Trading," Papers 2210.15448, arXiv.org, revised Sep 2023.
    20. Felix Drinkall & Janet B. Pierrehumbert & Stefan Zohren, 2024. "Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMs," Papers 2407.17624, arXiv.org, revised Jan 2025.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2305.08524. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.