IDEAS home Printed from https://ideas.repec.org/p/msh/ebswps/2018-4.html
   My bibliography  Save this paper

Anomaly detection in streaming nonstationary temporal data

Author

Listed:
  • Priyanga Dilini Talagala
  • Rob J Hyndman
  • Kate Smith-Miles
  • Sevvandi Kandanaarachchi
  • Mario A Munoz

Abstract

This article proposes a framework that provides early detection of anomalous series within a large collection of non-stationary streaming time series data. We define an anomaly as an observation that is very unlikely given the recent distribution of a given system. The proposed framework first forecasts a boundary for the system's typical behavior using extreme value theory. Then a sliding window is used to test for anomalous series within a newly arrived collection of series. The model uses time series features as inputs, and a density-based comparison to detect any significant changes in the distribution of the features. Using various synthetic and real world datasets, we demonstrate the wide applicability and usefulness of our proposed framework. We show that the proposed algorithm can work well in the presence of noisy non-stationarity data within multiple classes of time series. This framework is implemented in the open source R package oddstream. R code and data are available in the supplementary materials.

Suggested Citation

  • Priyanga Dilini Talagala & Rob J Hyndman & Kate Smith-Miles & Sevvandi Kandanaarachchi & Mario A Munoz, 2018. "Anomaly detection in streaming nonstationary temporal data," Monash Econometrics and Business Statistics Working Papers 4/18, Monash University, Department of Econometrics and Business Statistics.
  • Handle: RePEc:msh:ebswps:2018-4
    as

    Download full text from publisher

    File URL: https://www.monash.edu/business/ebs/research/publications/ebs/wp04-2018.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Pierre Perron & Gabriel Rodríguez, 2003. "Searching For Additive Outliers In Nonstationary Time Series," Journal of Time Series Analysis, Wiley Blackwell, vol. 24(2), pages 193-220, March.
    2. Kang, Yanfei & Hyndman, Rob J. & Smith-Miles, Kate, 2017. "Visualising forecasting algorithm performance using time series instance spaces," International Journal of Forecasting, Elsevier, vol. 33(2), pages 345-358.
    3. Wickham, Hadley, 2007. "Reshaping Data with the reshape Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 21(i12).
    4. Peter Burridge & A. M. Robert Taylor, 2006. "Additive Outlier Detection Via Extreme‐Value Theory," Journal of Time Series Analysis, Wiley Blackwell, vol. 27(5), pages 685-701, September.
    5. David E. Rapach & Jack K. Strauss, 2008. "Structural breaks and GARCH models of exchange rate volatility," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 23(1), pages 65-90.
    6. Anderson, N. H. & Hall, P. & Titterington, D. M., 1994. "Two-Sample Test Statistics for Measuring Discrepancies Between Two Multivariate Probability Density Functions Using Kernel-Based Density Estimates," Journal of Multivariate Analysis, Elsevier, vol. 50(1), pages 41-54, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Priyanga Dilini Talagala & Rob J Hyndman & Catherine Leigh & Kerrie Mengersen & Kate Smith-Miles, 2019. "A Feature-Based Framework for Detecting Technical Outliers in Water-Quality Data from In Situ Sensors," Monash Econometrics and Business Statistics Working Papers 1/19, Monash University, Department of Econometrics and Business Statistics.
    2. Sevvandi Kandanaarachchi & Mario A Munoz & Rob J Hyndman & Kate Smith-Miles, 2018. "On normalization and algorithm selection for unsupervised outlier detection," Monash Econometrics and Business Statistics Working Papers 16/18, Monash University, Department of Econometrics and Business Statistics.
    3. Miguel Flores & Salvador Naya & Rubén Fernández-Casal & Sonia Zaragoza & Paula Raña & Javier Tarrío-Saavedra, 2020. "Constructing a Control Chart Using Functional Data," Mathematics, MDPI, vol. 8(1), pages 1-26, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gabriel Rodriguez & Dionisio Ramirez, 2014. "A Note on the Size of the ADF Test with Additive Outliers and Fractional Errors. A Reappraisal about the (Non)Stationarity of the Latin-American Inflation Series," Revista Economía, Fondo Editorial - Pontificia Universidad Católica del Perú, vol. 37(73), pages 113-132.
    2. Priyanga Dilini Talagala & Rob J Hyndman & Kate Smith-Miles, 2019. "Anomaly Detection in High Dimensional Data," Monash Econometrics and Business Statistics Working Papers 20/19, Monash University, Department of Econometrics and Business Statistics.
    3. Harvey, David I. & Leybourne, Stephen J. & Taylor, A.M. Robert, 2010. "Robust methods for detecting multiple level breaks in autocorrelated time series," Journal of Econometrics, Elsevier, vol. 157(2), pages 342-358, August.
    4. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    5. Sam Astill & David I. Harvey & A. M. Robert Taylor, 2013. "A bootstrap test for additive outliers in non-stationary time series," Journal of Time Series Analysis, Wiley Blackwell, vol. 34(4), pages 454-465, July.
    6. Gabriel Rodriguez & Dionisio Ramirez, 2013. "A comparison between Tau-d and the procedure TRAMO-SEATS is also included," Documentos de Trabajo / Working Papers 2013-355, Departamento de Economía - Pontificia Universidad Católica del Perú.
    7. Alanya-Beltran, Willy, 2022. "Unit roots in lower-bounded series with outliers," Economic Modelling, Elsevier, vol. 115(C).
    8. David I. Harvey & Stephen J. Leybourne & A. M. Robert Taylor, 2009. "Robust methods for detecting multiple level breaks in autocorrelated time series [Revised to become No. 10/01 above]," Discussion Papers 09/01, University of Nottingham, Granger Centre for Time Series Econometrics.
    9. Hillebrand, Eric & Schnabl, Gunther & Ulu, Yasemin, 2009. "Japanese foreign exchange intervention and the yen-to-dollar exchange rate: A simultaneous equations approach using realized volatility," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 19(3), pages 490-505, July.
    10. Smyl, Slawek, 2020. "A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting," International Journal of Forecasting, Elsevier, vol. 36(1), pages 75-85.
    11. Peter Fuleky & Carl S. Bonham & Qianxue Zhao, 2013. "Estimating Demand Elasticities in Non-Stationary Panels: The Case of Hawaii's Tourism Industry," Working Papers 201314, University of Hawaii at Manoa, Department of Economics.
    12. Berens, Tobias & Weiß, Gregor N.F. & Wied, Dominik, 2015. "Testing for structural breaks in correlations: Does it improve Value-at-Risk forecasting?," Journal of Empirical Finance, Elsevier, vol. 32(C), pages 135-152.
    13. Augustinus, Benno A. & Blum, Moshe & Citterio, Sandra & Gentili, Rodolfo & Helman, David & Nestel, David & Schaffner, Urs & Müller-Schärer, Heinz & Lensky, Itamar M., 2022. "Ground-truthing predictions of a demographic model driven by land surface temperatures with a weed biocontrol cage experiment," Ecological Modelling, Elsevier, vol. 466(C).
    14. Ewing, Bradley T. & Malik, Farooq, 2016. "Volatility spillovers between oil prices and the stock market under structural breaks," Global Finance Journal, Elsevier, vol. 29(C), pages 12-23.
    15. Niels Haldrup & Antonio Montañés & Andreu Sansó, 2004. "Testing for Additive Outliers in Seasonally Integrated Time Series," Economics Working Papers 2004-14, Department of Economics and Business Economics, Aarhus University.
    16. Alberto Humala, 2008. "South American disinflation and regime switches: unobserved volatility components?," Monetaria, CEMLA, vol. 0(3), pages 405-425, julio-sep.
    17. Julio Cesar Alonso Cifuentes & Jaime Andres Carabali, 2019. "Breve Tuturial para visualizar y Calcular Métricas de Redes (grafos) en R (para Económisas)," Icesi Economics Lecture Notes 18170, Universidad Icesi.
    18. Jean-David Fermanian & Dominique Guégan, 2021. "Fair learning with bagging," Documents de travail du Centre d'Economie de la Sorbonne 21034, Université Panthéon-Sorbonne (Paris 1), Centre d'Economie de la Sorbonne.
    19. Matei Demetrescu & Christoph Hanck & Robinson Kruse‐Becher, 2022. "Robust inference under time‐varying volatility: A real‐time evaluation of professional forecasters," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(5), pages 1010-1030, August.
    20. Malik, Farooq, 2021. "Volatility spillover between exchange rate and stock returns under volatility shifts," The Quarterly Review of Economics and Finance, Elsevier, vol. 80(C), pages 605-613.

    More about this item

    Keywords

    concept drift; extreme value theory; feature-based time series analysis; kernel-based density estimation; multivariate time series; outlier detection.;
    All these keywords.

    JEL classification:

    • C38 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Classification Methdos; Cluster Analysis; Principal Components; Factor Analysis
    • C55 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Large Data Sets: Modeling and Analysis
    • C60 - Mathematical and Quantitative Methods - - Mathematical Methods; Programming Models; Mathematical and Simulation Modeling - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:msh:ebswps:2018-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Professor Xibin Zhang (email available below). General contact details of provider: https://edirc.repec.org/data/dxmonau.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.