IDEAS home Printed from https://ideas.repec.org/a/spr/annopr/v349y2025i3d10.1007_s10479-025-06585-w.html
   My bibliography  Save this article

On using fuzzy clustering for detecting the number of states in Markov switching models

Author

Listed:
  • Edoardo Otranto

    (Sapienza University of Rome, and CRENoS)

  • Luca Scaffidi Domianello

    (University of Catania)

Abstract

An open problem of Markov switching models is identifying the number of states, generally fixed a priori; it is impossible to apply classical tests due to the issue of the nuisance parameters present only under the alternative hypothesis. In this work, we show, by Monte Carlo simulations, that fuzzy clustering is able to reproduce the parametric state inference derived from the Hamilton filter and that the typical indices used in clustering to determine the number of groups can be used to identify the number of states in this framework. The procedure is very simple to apply, considering that it is performed independently of the data generating process and that the indicators we use are available in most statistical packages. Furthermore, the proposed approach appears to be sufficiently robust to perturbations in the data generating processes. A final application of real data completes the analysis.

Suggested Citation

  • Edoardo Otranto & Luca Scaffidi Domianello, 2025. "On using fuzzy clustering for detecting the number of states in Markov switching models," Annals of Operations Research, Springer, vol. 349(3), pages 1855-1890, June.
    • Handle: RePEc:spr:annopr:v:349:y:2025:i:3:d:10.1007_s10479-025-06585-w
    DOI: 10.1007/s10479-025-06585-w
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10479-025-06585-w
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10479-025-06585-w?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Dimitrios Koutmos, 2020. "Market risk and Bitcoin returns," Annals of Operations Research, Springer, vol. 294(1), pages 453-477, November.
    2. Giampiero M. Gallo & Demetrio Lacava & Edoardo Otranto, 2020. "On Classifying the Effects of Policy Announcements on Volatility," Papers 2011.14094, arXiv.org, revised Feb 2021.
    3. Kiefer, Nicholas M, 1978. "Discrete Parameter Variation: Efficient Estimation of a Switching Regression Model," Econometrica, Econometric Society, vol. 46(2), pages 427-434, March.
    4. Marcelle Chauvet & James D. Hamilton, 2006. "Dating Business Cycle Turning Points," Contributions to Economic Analysis, in: Nonlinear Time Series Analysis of Business Cycles, pages 1-54, Emerald Group Publishing Limited.
    5. Peter R. Hansen & Asger Lunde & James M. Nason, 2011. "The Model Confidence Set," Econometrica, Econometric Society, vol. 79(2), pages 453-497, March.
    6. Zhongjun Qu & Fan Zhuo, 2021. "Likelihood Ratio-Based Tests for Markov Regime Switching," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 88(2), pages 937-968.
    7. Albert, James H & Chib, Siddhartha, 1993. "Bayes Inference via Gibbs Sampling of Autoregressive Time Series Subject to Markov Mean and Variance Shifts," Journal of Business & Economic Statistics, American Statistical Association, vol. 11(1), pages 1-15, January.
    8. Markus Haas, 2004. "A New Approach to Markov-Switching GARCH Models," Journal of Financial Econometrics, Oxford University Press, vol. 2(4), pages 493-530.
    9. Engel, Charles & Hamilton, James D, 1990. "Long Swings in the Dollar: Are They in the Data and Do Markets Know It?," American Economic Review, American Economic Association, vol. 80(4), pages 689-713, September.
    10. Cheung, Yin-Wong & Erlandsson, Ulf G., 2005. "Exchange Rates and Markov Switching Dynamics," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 314-320, July.
    11. Hamilton, James D, 1989. "A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle," Econometrica, Econometric Society, vol. 57(2), pages 357-384, March.
    12. Hansen, Bruce E, 1996. "Inference When a Nuisance Parameter Is Not Identified under the Null Hypothesis," Econometrica, Econometric Society, vol. 64(2), pages 413-430, March.
    13. Glosten, Lawrence R & Jagannathan, Ravi & Runkle, David E, 1993. "On the Relation between the Expected Value and the Volatility of the Nominal Excess Return on Stocks," Journal of Finance, American Finance Association, vol. 48(5), pages 1779-1801, December.
    14. Smith, Aaron & Naik, Prasad A. & Tsai, Chih-Ling, 2006. "Markov-switching model selection using Kullback-Leibler divergence," Journal of Econometrics, Elsevier, vol. 134(2), pages 553-577, October.
    15. Hamilton, James D., 1990. "Analysis of time series subject to changes in regime," Journal of Econometrics, Elsevier, vol. 45(1-2), pages 39-70.
    16. Garcia, Rene, 1998. "Asymptotic Null Distribution of the Likelihood Ratio Test in Markov Switching Models," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(3), pages 763-788, August.
    17. Hathaway, Richard J., 1986. "Another interpretation of the EM algorithm for mixture distributions," Statistics & Probability Letters, Elsevier, vol. 4(2), pages 53-56, March.
    18. Zacharias Psaradakis & Nicola Spagnolo, 2003. "On The Determination Of The Number Of Regimes In Markov‐Switching Autoregressive Models," Journal of Time Series Analysis, Wiley Blackwell, vol. 24(2), pages 237-252, March.
    19. Hansen, Bruce E, 1996. "Erratum: The Likelihood Ratio Test under Nonstandard Conditions: Testing the Markov Switching Model of GNP," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 11(2), pages 195-198, March-Apr.
    20. Zacharias Psaradakis & Nicola Spagnolo, 2006. "Joint Determination of the State Dimension and Autoregressive Order for Models with Markov Regime Switching," Journal of Time Series Analysis, Wiley Blackwell, vol. 27(5), pages 753-766, September.
    21. Perron, P, 1993. "Erratum [The Great Crash, the Oil Price Shock and the Unit Root Hypothesis]," Econometrica, Econometric Society, vol. 61(1), pages 248-249, January.
    22. Yanlin Shi, 2023. "A simulation study on the Markov regime-switching zero-drift GARCH model," Annals of Operations Research, Springer, vol. 330(1), pages 1-20, November.
    23. Giampiero M. Gallo & Edoardo Otranto, 2018. "Combining sharp and smooth transitions in volatility dynamics: a fuzzy regime approach," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 67(3), pages 549-573, April.
    24. Roy Cerqueti & Hayette Gatfaoui & Giulia Rotundo, 2024. "Correction: Resilience for financial networks under a multivariate GARCH model of stock index returns with multiple regimes," Annals of Operations Research, Springer, vol. 335(1), pages 637-637, April.
    25. Nelson, Daniel B, 1991. "Conditional Heteroskedasticity in Asset Returns: A New Approach," Econometrica, Econometric Society, vol. 59(2), pages 347-370, March.
    26. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    27. Robert E. McCulloch & Ruey S. Tsay, 1994. "Statistical Analysis Of Economic Time Series Via Markov Switching Models," Journal of Time Series Analysis, Wiley Blackwell, vol. 15(5), pages 523-539, September.
    28. Marine Carrasco & Liang Hu & Werner Ploberger, 2014. "Optimal Test for Markov Switching Parameters," Econometrica, Econometric Society, vol. 82(2), pages 765-784, March.
    29. Gallo, Giampiero M. & Otranto, Edoardo, 2015. "Forecasting realized volatility with changing average levels," International Journal of Forecasting, Elsevier, vol. 31(3), pages 620-634.
    30. George Kapetanios, 2001. "Model Selection in Threshold Models," Journal of Time Series Analysis, Wiley Blackwell, vol. 22(6), pages 733-754, November.
    31. Perron, Pierre, 1989. "The Great Crash, the Oil Price Shock, and the Unit Root Hypothesis," Econometrica, Econometric Society, vol. 57(6), pages 1361-1401, November.
    32. Degras, David & Ting, Chee-Ming & Ombao, Hernando, 2022. "Markov-switching state-space models with applications to neuroimaging," Computational Statistics & Data Analysis, Elsevier, vol. 174(C).
    33. Jin Seo Cho & Halbert White, 2007. "Testing for Regime Switching," Econometrica, Econometric Society, vol. 75(6), pages 1671-1720, November.
    34. Lawrence Hubert & Phipps Arabie, 1985. "Comparing partitions," Journal of Classification, Springer;The Classification Society, vol. 2(1), pages 193-218, December.
    35. Kim, Chang-Jin, 1994. "Dynamic linear models with Markov-switching," Journal of Econometrics, Elsevier, vol. 60(1-2), pages 1-22.
    36. Edoardo Otranto & Giampiero Gallo, 2002. "A Nonparametric Bayesian Approach To Detect The Number Of Regimes In Markov Switching Models," Econometric Reviews, Taylor & Francis Journals, vol. 21(4), pages 477-496.
    37. Chang-Jin Kim & Charles R. Nelson, 1999. "State-Space Models with Regime Switching: Classical and Gibbs-Sampling Approaches with Applications," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262112388, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. L. Scaffidi Domianello & E. Otranto, 2023. "On the relationship between Markov Switching inference and Fuzzy Clustering: A Monte Carlo evidence," Working Paper CRENoS 202304, Centre for North South Economic Research, University of Cagliari and Sassari, Sardinia.
    2. Masaru Chiba, 2023. "Robust and efficient specification tests in Markov-switching autoregressive models," Statistical Inference for Stochastic Processes, Springer, vol. 26(1), pages 99-137, April.
    3. Nan Li & Simon S. Kwok, 2021. "Jointly determining the state dimension and lag order for Markov‐switching vector autoregressive models," Journal of Time Series Analysis, Wiley Blackwell, vol. 42(4), pages 471-491, July.
    4. Chung-Ming Kuan, 2013. "Markov switching model (in Russian)," Quantile, Quantile, issue 11, pages 13-40, December.
    5. Giampiero M. Gallo & Edoardo Otranto, 2014. "Forecasting Realized Volatility with Changes of Regimes," Econometrics Working Papers Archive 2014_03, Universita' degli Studi di Firenze, Dipartimento di Statistica, Informatica, Applicazioni "G. Parenti", revised Feb 2014.
    6. Franses,Philip Hans & Dijk,Dick van & Opschoor,Anne, 2014. "Time Series Models for Business and Economic Forecasting," Cambridge Books, Cambridge University Press, number 9780521520911, November.
    7. Odendahl, Florens & Rossi, Barbara & Sekhposyan, Tatevik, 2023. "Evaluating forecast performance with state dependence," Journal of Econometrics, Elsevier, vol. 237(2).
    8. Chew Lian Chua & Sandy Suardi, 2005. "Is There a Unit Root in East-Asian Short-Term Interest Rates?," Melbourne Institute Working Paper Series wp2005n14, Melbourne Institute of Applied Economic and Social Research, The University of Melbourne.
    9. Smith, Aaron & Naik, Prasad A. & Tsai, Chih-Ling, 2006. "Markov-switching model selection using Kullback-Leibler divergence," Journal of Econometrics, Elsevier, vol. 134(2), pages 553-577, October.
    10. King, Daniel & Botha, Ferdi, 2015. "Modelling stock return volatility dynamics in selected African markets," Economic Modelling, Elsevier, vol. 45(C), pages 50-73.
    11. Gallo, Giampiero M. & Otranto, Edoardo, 2015. "Forecasting realized volatility with changing average levels," International Journal of Forecasting, Elsevier, vol. 31(3), pages 620-634.
    12. Gabriel Rodriguez-Rondon & Jean-Marie Dufour, 2024. "MSTest: An R-Package for Testing Markov Switching Models," Papers 2411.08188, arXiv.org.
    13. Hamilton, J.D., 2016. "Macroeconomic Regimes and Regime Shifts," Handbook of Macroeconomics, in: J. B. Taylor & Harald Uhlig (ed.), Handbook of Macroeconomics, edition 1, volume 2, chapter 0, pages 163-201, Elsevier.
    14. Yuan, Chunming, 2011. "The exchange rate and macroeconomic determinants: Time-varying transitional dynamics," The North American Journal of Economics and Finance, Elsevier, vol. 22(2), pages 197-220, August.
    15. Sean D. Campbell, 2002. "Specification Testing and Semiparametric Estimation of Regime Switching Models: An Examination of the US Short Term Interest Rate," Working Papers 2002-26, Brown University, Department of Economics.
    16. Massimo Guidolin, 2011. "Markov Switching Models in Empirical Finance," Advances in Econometrics, in: Missing Data Methods: Time-Series Methods and Applications, pages 1-86, Emerald Group Publishing Limited.
    17. Richard D. F. Harris & Murat Mazibas, 2022. "A component Markov regime‐switching autoregressive conditional range model," Bulletin of Economic Research, Wiley Blackwell, vol. 74(2), pages 650-683, April.
    18. Luca Scaffidi Domianello & Giampiero M. Gallo & Edoardo Otranto, 2024. "Smooth and Abrupt Dynamics in Financial Volatility: The MS‐MEM‐MIDAS," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 86(1), pages 21-43, February.
    19. Kim, Chang-Jin & Nelson, Charles R, 2001. "A Bayesian Approach to Testing for Markov-Switching in Univariate and Dynamic Factor Models," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 42(4), pages 989-1013, November.
    20. Maddalena Cavicchioli, 2015. "Likelihood Ratio Test and Information Criteria for Markov Switching Var Models: An Application to the Italian Macroeconomy," Italian Economic Journal: A Continuation of Rivista Italiana degli Economisti and Giornale degli Economisti, Springer;Società Italiana degli Economisti (Italian Economic Association), vol. 1(3), pages 315-332, November.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:annopr:v:349:y:2025:i:3:d:10.1007_s10479-025-06585-w. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.