For a Markov chain \(\{ X_{n} \}_{n=0}^\infty\) we consider whether the limit \(\lim_{ n \to \infty }p_{i,j}(n)\) exists for all \(j\) independently of \(i\), whether it is a distribution, and whether the chain visits \(j\) at \(n=\infty\). From state classification results we have the following result.
If state \(j\) is transient then \(\lim_{ n \to \infty }p_{i,j}(n)=0\) for all \(i\).
If the limiting distribution \(\pi=(\pi_{j}=\lim_{ n \to \infty }p_{i,j}(n),~j\in S)\) exists, it must be stationary. More formally we have the following result.
For all \(j\), let \(p_{i,j}(n)\to \pi_{j}\) and \(\pi=(\pi_{j},~j \in S)\) be a distribution, that is \(\pi_{j} \geq 0\), \(\sum_{j}\pi_{j}=1\). Then, \(\pi\) is stationary.
For periodic states \(\lim_{ n \to \infty }p_{i,j}(n)\) might not exist.
The following theorem for ergodic (aperiodic non-null persistent) states is an important result.
Let \((X_{n})\) be an irreducible non-null persistent Markov chain with stationary distributions \(\pi\) such that \(\pi=\pi P\), we have the following results: 1. If the chain is aperiodic then \(\lim_{ n \to \infty }p_{i,j}(n)=\pi_{j}=\frac{1}{\mu_{j}}\) for all \(j \in S\). 2. If the chain is periodic with period \(d\), then for all \(i,j\in S\) there exists an integer \(r\) such that \(0\leq r<d\) and \(p_{i,j}(n)=0\) unless \(n=md+r\) for some \(m\geq 0\) and \(\lim_{ m \to \infty }p_{i,j}(md+r)=d\cdot \pi_{j}= \frac{d}{\mu_{j}}\).
For null persistent irreducible chains these results hold with \(\mu_{j}=\infty\) i.e. \(\lim_{ n \to \infty }p_{i,j}(n)=0\) in aperiodic case and \(\lim_{ m \to \infty }p_{i,j}(md+r)=0\) in periodic case.
We discuss the result in the following points:
In ergodic case, \(p_{i,j}(n)\to \pi_{j}= \frac{1}{\mu_{j}}\) and the chain forgets its origin: \[\mathbb{P}(X_{n}=j)=\sum_{i \in S}\mathbb{P}(X_{n}=j|X_{0}=i)\mathbb{P}(X_{0}=i)=\sum_{i}p_{i,j}(n)\cdot \mathbb{P}(X_{0}=i)\stackrel{n \to \infty}{\to}\pi_{j}\sum_{i}\mathbb{P}(X_{0}=i)=\pi_{j}.\]Here we used the Dominated Convergence Theorem to justify exchanging summation and limits: \[\left| \sum_{i}p_{i,n}(n)\mathbb{P}(X_{0}=i) \right|\leq \sum_{i}\mathbb{P}(X_{0}=i)=1.\]
A finite irreducible aperiodic Markov chain is always non-null persistent (see previous results) so that the stationary distribution always exists and by the ergodic theorem, \(\lim_{ n \to \infty }p_{i,j}(n)=\pi p_{j}=\frac{1}{\mu_{j}}\).
An irreducible aperiodic Markov chain has: stationary distribution \(\iff\) non-null persistent \(\iff\) there exists \(\lim_{ n \to \infty }p_{i,j}(n)=\pi_{j}\) and \(\pi=(\pi_{j},~j \in S)\) is a probability distribution (ergodic chain!).