Markov Chains

Markov chains are widely used discrete time discrete state space stochastic processes. In this chapter we study in sufficient detail the classication of Markov chains which is the first step in analyzing a Markov chain. The basic denitions related to Markov chains are given in the first section. The higher-step transition probabilities and related results are given in Sect. 2. Section 3 deals with generation of realizations (sample paths) of specied length of a Markov chain. This section also contains a brief discussion on the maximum likelihood estimation of the transition probability matrix. The topic of classfiication of the states of a Markov chain, which is essential for the study of long-run behaviour of the Markov chains, is discussed Sects. 4 and 5. An extended discussion and some important results on first passage distributions are given in Sect. 6. Computation of probabilities of absorption into recurring classes, when set of transient states is finite, is also considered in this section. In Section 7, the concept of ‘periodicity’ of states is discussed in detail. In all the sections, the concepts and results are illustrated with computations using R codes given in Sect. 8. These R programs are useful (i) to obtain higher-step transition probabilities (ii) to obtain a finite dimensional distribution (iii) to generate a realization of specified length of a Markov chain (iv) to compute the maximum likelihood estimate of a transition probability matrix (v) to classify the states as persistent or transient (vi) to compute first passage distributions and (vii) to find the period of various states.