IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2601.05716.html

When the Rules Change: Adaptive Signal Extraction via Kalman Filtering and Markov-Switching Regimes

Author

Listed:
  • Sungwoo Kang

Abstract

Most empirical microstructure research assumes that order flow--return parameters are constant, yet these relationships shift substantially across market regimes. Combining adaptive Kalman filtering, Markov-switching regime identification, and asymmetric response estimation, we characterize regime-dependent investor behavior in the Korean stock market during 2020--2024 using daily transaction data disaggregated by investor type. Three principal findings emerge: foreign investor predictive power increases several-fold during crisis periods relative to bull markets; individual investors chase momentum asymmetrically, reacting far more strongly to positive than to negative shocks; and independent information-theoretic validation corroborates both patterns. Rigorous out-of-sample testing reveals that these in-sample regularities do not generalize reliably, underscoring the need for proper validation methodology in microstructure research.

Suggested Citation

  • Sungwoo Kang, 2026. "When the Rules Change: Adaptive Signal Extraction via Kalman Filtering and Markov-Switching Regimes," Papers 2601.05716, arXiv.org, revised Feb 2026.
    • Handle: RePEc:arx:papers:2601.05716
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2601.05716
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Hasbrouck, Joel, 1991. "Measuring the Information Content of Stock Trades," Journal of Finance, American Finance Association, vol. 46(1), pages 179-207, March.
    2. John M. Griffin & Jeffrey H. Harris & Selim Topaloglu, 2003. "The Dynamics of Institutional and Individual Trading," Journal of Finance, American Finance Association, vol. 58(6), pages 2285-2320, December.
    3. Mark Grinblatt & Matti Keloharju, 2001. "What Makes Investors Trade?," Journal of Finance, American Finance Association, vol. 56(2), pages 589-616, April.
    4. John Y. Campbell & Samuel B. Thompson, 2008. "Predicting Excess Stock Returns Out of Sample: Can Anything Beat the Historical Average?," The Review of Financial Studies, Society for Financial Studies, vol. 21(4), pages 1509-1531, July.
    5. Andrew Ang & Geert Bekaert, 2002. "International Asset Allocation With Regime Shifts," The Review of Financial Studies, Society for Financial Studies, vol. 15(4), pages 1137-1187.
    6. Glosten, Lawrence R. & Milgrom, Paul R., 1985. "Bid, ask and transaction prices in a specialist market with heterogeneously informed traders," Journal of Financial Economics, Elsevier, vol. 14(1), pages 71-100, March.
    7. Shefrin, Hersh & Statman, Meir, 1985. "The Disposition to Sell Winners Too Early and Ride Losers Too Long: Theory and Evidence," Journal of Finance, American Finance Association, vol. 40(3), pages 777-790, July.
    8. R. David Mclean & Jeffrey Pontiff, 2016. "Does Academic Research Destroy Stock Return Predictability?," Journal of Finance, American Finance Association, vol. 71(1), pages 5-32, February.
    9. Kyle, Albert S, 1985. "Continuous Auctions and Insider Trading," Econometrica, Econometric Society, vol. 53(6), pages 1315-1335, November.
    10. Tarun Chordia & Richard Roll & Avanidhar Subrahmanyam, 2001. "Market Liquidity and Trading Activity," Journal of Finance, American Finance Association, vol. 56(2), pages 501-530, April.
    11. Barber, Brad M. & Odean, Terrance, 2013. "The Behavior of Individual Investors," Handbook of the Economics of Finance, in: G.M. Constantinides & M. Harris & R. M. Stulz (ed.), Handbook of the Economics of Finance, volume 2, chapter 0, pages 1533-1570, Elsevier.
    12. Daniel Kahneman & Amos Tversky, 2013. "Prospect Theory: An Analysis of Decision Under Risk," World Scientific Book Chapters, in: Leonard C MacLean & William T Ziemba (ed.), HANDBOOK OF THE FUNDAMENTALS OF FINANCIAL DECISION MAKING Part I, chapter 6, pages 99-127, World Scientific Publishing Co. Pte. Ltd..
    13. Ivo Welch & Amit Goyal, 2008. "A Comprehensive Look at The Empirical Performance of Equity Premium Prediction," The Review of Financial Studies, Society for Financial Studies, vol. 21(4), pages 1455-1508, July.
    14. Ron Kaniel & Gideon Saar & Sheridan Titman, 2008. "Individual Investor Trading and Stock Returns," Journal of Finance, American Finance Association, vol. 63(1), pages 273-310, February.
    15. repec:bla:jfinan:v:53:y:1998:i:5:p:1775-1798 is not listed on IDEAS
    16. Brad M. Barber & Terrance Odean, 2008. "All That Glitters: The Effect of Attention and News on the Buying Behavior of Individual and Institutional Investors," The Review of Financial Studies, Society for Financial Studies, vol. 21(2), pages 785-818, April.
    17. Hamilton, James D, 1989. "A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle," Econometrica, Econometric Society, vol. 57(2), pages 357-384, March.
    18. Massimo Guidolin & Allan Timmermann, 2006. "An econometric model of nonlinear dynamics in the joint distribution of stock and bond returns," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 21(1), pages 1-22, January.
    19. Lee, Charles M C & Ready, Mark J, 1991. "Inferring Trade Direction from Intraday Data," Journal of Finance, American Finance Association, vol. 46(2), pages 733-746, June.
    20. Kim, Woochan & Wei, Shang-Jin, 2002. "Foreign portfolio investors before and during a crisis," Journal of International Economics, Elsevier, vol. 56(1), pages 77-96, January.
    21. Mark Grinblatt & Matti Keloharju, 2000. "What Makes Investors Trade?," Yale School of Management Working Papers ysm146, Yale School of Management, revised 01 Nov 2001.
    22. Durbin, James & Koopman, Siem Jan, 2012. "Time Series Analysis by State Space Methods," OUP Catalogue, Oxford University Press, edition 2, number 9780199641178.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Choi, Darwin, 2019. "Disposition sales and stock market liquidity," Journal of Financial Markets, Elsevier, vol. 45(C), pages 19-36.
    2. Barber, Brad M. & Odean, Terrance, 2013. "The Behavior of Individual Investors," Handbook of the Economics of Finance, in: G.M. Constantinides & M. Harris & R. M. Stulz (ed.), Handbook of the Economics of Finance, volume 2, chapter 0, pages 1533-1570, Elsevier.
    3. Fotini Economou & Konstantinos Gavriilidis & Bartosz Gebka & Vasileios Kallinterakis, 2022. "Feedback trading: a review of theory and empirical evidence," Review of Behavioral Finance, Emerald Group Publishing Limited, vol. 15(4), pages 429-476, February.
    4. Peress, Joel & Schmidt, Daniel, 2021. "Noise traders incarnate: Describing a realistic noise trading process," Journal of Financial Markets, Elsevier, vol. 54(C).
    5. Danny Lo, 2015. "Essays in Market Microstructure and Investor Trading," PhD Thesis, Finance Discipline Group, UTS Business School, University of Technology, Sydney, number 4-2015, January-A.
    6. Lei, Qin & Wu, Guojun, 2005. "Time-varying informed and uninformed trading activities," Journal of Financial Markets, Elsevier, vol. 8(2), pages 153-181, May.
    7. Cui, Xudong & Gong, Pu & Liu, Tong, 2025. "The disposition effect and market volatility prediction," International Review of Financial Analysis, Elsevier, vol. 108(PB).
    8. Rzayev, Khaladdin & Ibikunle, Gbenga, 2019. "A state-space modeling of the information content of trading volume," Journal of Financial Markets, Elsevier, vol. 46(C).
    9. Barber, Brad M. & Odean, Terrance & Zhu, Ning, 2009. "Systematic noise," Journal of Financial Markets, Elsevier, vol. 12(4), pages 547-569, November.
    10. Sankaraguruswamy, Srinivasan & Shen, Jianfeng & Yamada, Takeshi, 2013. "The relationship between the frequency of news release and the information asymmetry: The role of uninformed trading," Journal of Banking & Finance, Elsevier, vol. 37(11), pages 4134-4143.
    11. Danny Lo, 2015. "Essays in Market Microstructure and Investor Trading," PhD Thesis, Finance Discipline Group, UTS Business School, University of Technology, Sydney, number 22, July-Dece.
    12. Campbell, John Y. & Ramadorai, Tarun & Schwartz, Allie, 2009. "Caught on tape: Institutional trading, stock returns, and earnings announcements," Journal of Financial Economics, Elsevier, vol. 92(1), pages 66-91, April.
    13. Stoffman, Noah, 2014. "Who trades with whom? Individuals, institutions, and returns," Journal of Financial Markets, Elsevier, vol. 21(C), pages 50-75.
    14. Agudelo, Diego A. & Byder, James & Yepes-Henao, Paula, 2019. "Performance and informed trading. Comparing foreigners, institutions and individuals in an emerging stock market," Journal of International Money and Finance, Elsevier, vol. 90(C), pages 187-203.
    15. Cao, Ji & Muhl, Stefan & Rieger, Marc Oliver & Chen, Hung-Ling, 2023. "Sign matters: Stock-movement-based trading decisions of individual investors," Journal of Banking & Finance, Elsevier, vol. 148(C).
    16. Sungwoo Kang, 2026. "The Limits of Complexity: Why Feature Engineering Beats Deep Learning in Investor Flow Prediction," Papers 2601.07131, arXiv.org.
    17. Yue Zhao & Difang Wan, 2018. "Institutional high frequency trading and price discovery: Evidence from an emerging commodity futures market," Journal of Futures Markets, John Wiley & Sons, Ltd., vol. 38(2), pages 243-270, February.
    18. Jakusch, Sven Thorsten, 2017. "On the applicability of maximum likelihood methods: From experimental to financial data," SAFE Working Paper Series 148, Leibniz Institute for Financial Research SAFE, revised 2017.
    19. Francisco Gomes & Michael Haliassos & Tarun Ramadorai, 2021. "Household Finance," Journal of Economic Literature, American Economic Association, vol. 59(3), pages 919-1000, September.
    20. Weber, Martin & Mueller-Dethard, Jan, 2020. "The Portfolio Composition Effect," CEPR Discussion Papers 15012, C.E.P.R. Discussion Papers.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2601.05716. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.