IDEAS home Printed from https://ideas.repec.org/p/hal/wpaper/hal-03777995.html
   My bibliography  Save this paper

Anomaly Detection on Financial Time Series by Principal Component Analysis and Neural Networks

Author

Listed:
  • Stéphane Crépey

    (LPSM (UMR_8001) - Laboratoire de Probabilités, Statistique et Modélisation - SU - Sorbonne Université - CNRS - Centre National de la Recherche Scientifique - UPCité - Université Paris Cité, UPCité - Université Paris Cité)

  • Lehdili Noureddine

    (Natixis)

  • Nisrine Madhar

    (LPSM (UMR_8001) - Laboratoire de Probabilités, Statistique et Modélisation - SU - Sorbonne Université - CNRS - Centre National de la Recherche Scientifique - UPCité - Université Paris Cité, UPCité - Université Paris Cité, Natixis)

  • Maud Thomas

    (LPSM (UMR_8001) - Laboratoire de Probabilités, Statistique et Modélisation - SU - Sorbonne Université - CNRS - Centre National de la Recherche Scientifique - UPCité - Université Paris Cité, SU - Sorbonne Université)

Abstract

A major concern when dealing with financial time series involving a wide variety of market risk factors is the presence of anomalies. These induce a miscalibration of the models used to quantify and manage risk, resulting in potential erroneous risk measures. We propose an approach that aims to improve anomaly detection in financial time series, overcoming most of the inherent difficulties. Valuable features are extracted from the time series by compressing and reconstructing the data through principal component analysis. We then define an anomaly score using a feedforward neural network. A time series is considered to be contaminated when its anomaly score exceeds a given cutoff value. This cutoff value is not a hand-set parameter but rather is calibrated as a neural network parameter throughout the minimization of a customized loss function. The efficiency of the proposed approach compared to several well-known anomaly detection algorithms is numerically demonstrated on both synthetic and real data sets, with high and stable performance being achieved with the PCA NN approach. We show that value-at-risk estimation errors are reduced when the proposed anomaly detection model is used with a basic imputation approach to correct the anomaly.

Suggested Citation

  • Stéphane Crépey & Lehdili Noureddine & Nisrine Madhar & Maud Thomas, 2022. "Anomaly Detection on Financial Time Series by Principal Component Analysis and Neural Networks," Working Papers hal-03777995, HAL.
  • Handle: RePEc:hal:wpaper:hal-03777995
    Note: View the original document on HAL open archive server: https://hal.science/hal-03777995v3
    as

    Download full text from publisher

    File URL: https://hal.science/hal-03777995v3/document
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Erdinc Akyildirim & Matteo Gambara & Josef Teichmann & Syang Zhou, 2022. "Applications of Signature Methods to Market Anomaly Detection," Papers 2201.02441, arXiv.org, revised Feb 2022.
    2. Pierre Henry-Labordere, 2019. "(Martingale) Optimal Transport And Anomaly Detection With Neural Networks: A Primal-dual Algorithm," Papers 1904.04546, arXiv.org, revised Apr 2019.
    3. Douglas M. Hawkins, 1980. "Critical Values for Identifying Outliers," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 29(1), pages 95-96, March.
    4. Takaya Saito & Marc Rehmsmeier, 2015. "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets," PLOS ONE, Public Library of Science, vol. 10(3), pages 1-21, March.
    5. Yufeng Yu & Yuelong Zhu & Shijin Li & Dingsheng Wan, 2014. "Time Series Outlier Detection Based on Sliding Window Prediction," Mathematical Problems in Engineering, Hindawi, vol. 2014, pages 1-14, October.
    6. Pierre Henry-Labordère, 2019. "(Martingale) Optimal Transport And Anomaly Detection With Neural Networks: A Primal-Dual Algorithm," Working Papers hal-02095222, HAL.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. St'ephane Cr'epey & Lehdili Noureddine & Nisrine Madhar & Maud Thomas, 2022. "Anomaly Detection on Financial Time Series by Principal Component Analysis and Neural Networks," Papers 2209.11686, arXiv.org, revised Oct 2022.
    2. Hans Buhler & Blanka Horvath & Terry Lyons & Imanol Perez Arribas & Ben Wood, 2020. "A Data-driven Market Simulator for Small Data Environments," Papers 2006.14498, arXiv.org.
    3. Ariel Neufeld & Julian Sester, 2021. "Model-free price bounds under dynamic option trading," Papers 2101.01024, arXiv.org, revised Jul 2021.
    4. Tommaso Barbariol & Enrico Feltresi & Gian Antonio Susto, 2020. "Self-Diagnosis of Multiphase Flow Meters through Machine Learning-Based Anomaly Detection," Energies, MDPI, vol. 13(12), pages 1-24, June.
    5. Ariel Neufeld & Julian Sester & Daiying Yin, 2022. "Detecting data-driven robust statistical arbitrage strategies with deep neural networks," Papers 2203.03179, arXiv.org, revised Feb 2024.
    6. Joshua Zoen-Git Hiew & Tongseok Lim & Brendan Pass & Marcelo Cruz de Souza, 2023. "Geometry of vectorial martingale optimal transport and robust option pricing," Papers 2309.04947, arXiv.org, revised Sep 2023.
    7. Christopher J Greenwood & George J Youssef & Primrose Letcher & Jacqui A Macdonald & Lauryn J Hagg & Ann Sanson & Jenn Mcintosh & Delyse M Hutchinson & John W Toumbourou & Matthew Fuller-Tyszkiewicz &, 2020. "A comparison of penalised regression methods for informing the selection of predictive markers," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-14, November.
    8. Le, Hong Hanh & Viviani, Jean-Laurent, 2018. "Predicting bank failure: An improvement by implementing a machine-learning approach to classical financial ratios," Research in International Business and Finance, Elsevier, vol. 44(C), pages 16-25.
    9. João Chang Junior & Fábio Binuesa & Luiz Fernando Caneo & Aida Luiza Ribeiro Turquetto & Elisandra Cristina Trevisan Calvo Arita & Aline Cristina Barbosa & Alfredo Manoel da Silva Fernandes & Evelinda, 2020. "Improving preoperative risk-of-death prediction in surgery congenital heart defects using artificial intelligence model: A pilot study," PLOS ONE, Public Library of Science, vol. 15(9), pages 1-21, September.
    10. Arthur De Sá Ferreira & Ney Meziat-Filho & Ana Paula Antunes Ferreira, 2021. "Double threshold receiver operating characteristic plot for three-modal continuous predictors," Computational Statistics, Springer, vol. 36(3), pages 2231-2245, September.
    11. Masabho P Milali & Samson S Kiware & Nicodem J Govella & Fredros Okumu & Naveen Bansal & Serdar Bozdag & Jacques D Charlwood & Marta F Maia & Sheila B Ogoma & Floyd E Dowell & George F Corliss & Maggy, 2020. "An autoencoder and artificial neural network-based method to estimate parity status of wild mosquitoes from near-infrared spectra," PLOS ONE, Public Library of Science, vol. 15(6), pages 1-16, June.
    12. Kajal Lahiri & Cheng Yang, 2023. "ROC and PRC Approaches to Evaluate Recession Forecasts," Journal of Business Cycle Research, Springer;Centre for International Research on Economic Tendency Surveys (CIRET), vol. 19(2), pages 119-148, September.
    13. Tzu-Hsuan Lin & Jehn-Ruey Jiang, 2021. "Credit Card Fraud Detection with Autoencoder and Probabilistic Random Forest," Mathematics, MDPI, vol. 9(21), pages 1-16, October.
    14. Francesca Ieva & Anna Maria Paganoni, 2020. "Component-wise outlier detection methods for robustifying multivariate functional samples," Statistical Papers, Springer, vol. 61(2), pages 595-614, April.
    15. Alfred Krzywicki & David Muchlinski & Benjamin E. Goldsmith & Arcot Sowmya, 2022. "From academia to policy makers: a methodology for real-time forecasting of infrequent events," Journal of Computational Social Science, Springer, vol. 5(2), pages 1489-1510, November.
    16. Andrzej Chmielowiec, 2021. "Algorithm for error-free determination of the variance of all contiguous subsequences and fixed-length contiguous subsequences for a sequence of industrial measurement data," Computational Statistics, Springer, vol. 36(4), pages 2813-2840, December.
    17. Marco Due~nas & V'ictor Ortiz & Massimo Riccaboni & Francesco Serti, 2021. "Assessing the Impact of COVID-19 on Trade: a Machine Learning Counterfactual Analysis," Papers 2104.04570, arXiv.org.
    18. Michael Kupper & Max Nendel & Alessandro Sgarabottolo, 2023. "Risk measures based on weak optimal transport," Papers 2312.05973, arXiv.org.
    19. Marc Chataigner & Stéphane Crépey & Jiang Pu, 2020. "Nowcasting Networks," Post-Print hal-03910123, HAL.
    20. Greco, Salvatore & Ishizaka, Alessio & Tasiou, Menelaos & Torrisi, Gianpiero, 2019. "Sigma-Mu efficiency analysis: A methodology for evaluating units through composite indicators," European Journal of Operational Research, Elsevier, vol. 278(3), pages 942-960.

    More about this item

    Keywords

    anomaly detection; financial time series; principal component analysis; neural network; density estimation; missing data; market risk; value at risk;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:hal:wpaper:hal-03777995. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: CCSD (email available below). General contact details of provider: https://hal.archives-ouvertes.fr/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.