Markov Process
1 Intuition
A Markov process is a stochastic process indexed by time whose future is independent of its past given the present, the Markov property. Markov processes describe memoryless real-world systems that evolve based only on their current state.
The general (and strong) Markov property is stated in Markov Property. The discrete-time, countable-state case is developed in Markov Chains and its sub-topics; this note concerns the general picture and continuous time.
2 Classification
Markov processes fall into four groups according to the time index and the state space:
| Countable state space | General state space | |
|---|---|---|
| Discrete time | Markov Chains | Markov chains on a measurable state space |
| Continuous time | Jump Processes | diffusions and other processes, e.g. Brownian Motion |
3 Transition semigroup
For a time-homogeneous process on a state space \(S\) the transition function is \(P_{t}(x,A)=\mathbb{P}(X_{t+s}\in A|X_{s}=x)\). The Markov property implies the Chapman-Kolmogorov equations \[ P_{t+s}(x,A)=\int_{S}P_{s}(x,dy)\,P_{t}(y,A), \] so that \((P_{t})_{t\geq0}\) is a semigroup of operators acting on functions by \(P_{t}f(x)=\mathbb{E}[f(X_{t})|X_{0}=x]\). Differentiating this semigroup at \(t=0\) gives the Infinitesimal Generator, and the evolution of \(P_{t}\) and of the law of \(X_{t}\) is described by the Kolmogorov Equations (backward and forward).
4 References
- L. C. G. Rogers and D. Williams, Diffusions, Markov Processes and Martingales, Volume 1.
- J. R. Norris, Markov Chains, Cambridge University Press, 1997.