Continuous Time Markov Chains

This chapter is concerned with the general theory of continuous time discrete state space Markov processes. These processes are also known as Markov pure jump processes or continuous time Markov chains. In Sect. 2, the probabilistic structure of a continuous time Markov chain is discussed. It is proved that the holding times (sojourn times) in the states have exponential distributions. The concept of an embedded Markov chain is also introduced. In Sect. 3, some properties of the transition functions are established. The important concept of the infinitesimal generator of a continuous time Markov chain and its relation to the transition probability matrix of the embedded Markov chain are discussed. Section 5 is concerned with the solutions to Kolmogorov forward and backward equations and also some methods to compute transition functions numerically. In Sect. 6, the long-run behavior of continuous time Markov chains, balance equations and steady-state solutions are discussed. Section 7 contains R codes for computation of transition probability matrix of the embedded chain, for obtaining realizations from a continuous time Markov chain, for computing transition probability function and for computing the stationary and long-run distributions.