1 Probability Measure Theory
1.1 Probability Spaces
A probability space is a triple \((\Omega, \mathcal{F}, \mathbb{P})\) consisting of a set of outcomes of a random experiment \(\Omega\) - known as a sample space - along with a collection \(\mathcal{F}\) of measurable subsets (i.e. events) of \(\Omega\) - known as a \(\sigma\)-algebra - and a mapping \(\mathbb{P}:\mathcal{F}\to [0,1]\) assigning each event a probability - known as a **probability measure.
A \(\sigma\)-algebra \(\mathcal{F}\) is a collection of subspaces of \(\Omega\) satisfying the following properties: 1. Non-empty. \(\Omega \in \mathcal{F}\). 2. Closed under compliments. \(A\in \mathcal{F} \implies A^c \in \mathcal{F}\). 3. Closed under countable unions. \(A_{i}\in\mathcal{F}\) for countable \(i\) \(\implies \cup_{i}A_{i}\in\mathcal{F}\).
We say that any set \(A \in \mathcal{F}\) is measurable. We can show that for a (possibly uncountable) collection of \(\sigma\)-algebras \(\mathcal{F}_{i}\) with index set \(i\in I\) over sample space \(\Omega\) the intersection \(\cap_{i}\mathcal{F}_{i}\) is also a \(\sigma\)-algebra. This important result allows us to define the following.
For a collection \(\mathcal{A}\subseteq 2^\Omega\) of subsets of \(\Omega\) there is a unique smallest \(\sigma\)-algebra on \(\Omega\) containing \(\mathcal{A}\), known as the \(\sigma\)-algebra generated by \(\mathcal{A}\) defined as \[ \sigma(\mathcal{A})=\bigcap \{ \mathcal{F}:\mathcal{F}\text{ is a }\sigma \text{-algebra over }\Omega~s.t.~\mathcal{A}\subseteq \mathcal{F} \}. \]
The pair \((\Omega, \mathcal{F})\) of a sample space with a \(\sigma\)-algebra is known as a measurable space. Intuitively, we have taken our \(\Omega\) and simply made a collection of every single subset of \(\Omega\) that satisfies the conditions ensuring they behave correctly when we proceed to define our measure function.
A probability measure \(\mathbb{P}\) on measurable space \((\Omega,\mathcal{F})\) is a mapping \(\mathbb{P}:\mathcal{F}\mapsto [0,1]\) satisfying the following properties: 1. Non-negative. \(\mathbb{P}(A)\geq \mathbb{P}(\emptyset)=0\). 2. \(\mathbb{P}(\Omega)=1\); 3. Countable Additivity. For countable disjoint \(A_{i}\in\mathcal{F}\) then \(\mathbb{P}(\cup_{i}A_{i})=\sum_{i}\mathbb{P}(A_{i})\).
The triple \((\Omega, \mathcal{F}, \mathbb{P})\) of a measurable space with a defined probability measure is known as a probability space and several helpful properties which we summarize in the following theorem.
For a probability space \((\Omega, \mathcal{F}, \mathbb{P})\) the following results hold: 1. Monotonicity. \(A\subset B,~A,B\in \mathcal{F}\implies \mathbb{P}(A)\leq \mathbb{P}(B)\). 2. Rule of Addition. \(A,B \in \mathcal{F}\implies \mathbb{P}(A \cup B)+\mathbb{P}(A\cap B)=\mathbb{P}(A)+\mathbb{P}(B)\). 3. Subadditivity. \(\{ A_{n} \}_{n\in\mathbb{N}}\subseteq \mathcal{F}\implies P(\cup_{n=1}^\infty A_{n})\leq \sum_{n=1}^\infty \mathbb{P}(A_{n})\). 4. Continuity from below. \(A_{i}\uparrow A\) (i.e. \(A_1 \subset A_2 \subset\dots~\&~\bigcup_{i}A_{i}=A\)) \(\implies \mathbb{P}(A_{i})\uparrow \mathbb{P}(A)\). 5. Continuity from above. \(A_{i}\downarrow A\) (i.e. \(A_{1}\supset A_{2}\supset \dots ~\&~ \bigcap_{i}A_{i}=A\)) \(\implies \mathbb{P}(A_{i})\downarrow \mathbb{P}(A)\).
Proof: For (1) we have \(B=A\cup(B\setminus A)\) disjoint and so \(\mathbb{P}(B)=\mathbb{P}(A)+\mathbb{P}(B\setminus A)\geq \mathbb{P}(A)\). For (2) we have that \(A\cup B = (A\setminus B)\cup (B\setminus A)\cup (A \cap B)\) where all RH terms are disjoint. Therefore \[ \begin{align}\mathbb{P}(A\cup B)+\mathbb{P}(A\cap B) & = \underbrace{\mathbb{P}(A\setminus B)+\mathbb{P}(A \cap B)}_{\text{disjoint}}+\underbrace{\mathbb{P}(B\setminus A)+\mathbb{P}(A \cap B)}_{\text{disjoint}} \\ & = \mathbb{P}((A\setminus B)\cup (A\cap B)) + \mathbb{P}((B\setminus A)\cup(A \cap B)) \\ & =\mathbb{P}(A) + \mathbb{P}(B).\end{align} \] Result (3) follows by induction of the methodology from (2). For (4) define \(\{ B_{n} \}\) such that \(B_{n}=A_{n}\setminus A_{n-1}\) disjoint such that \(\cup_{n}B_{n}=\cup_{n}A_{n}=A_{n}\) and hence \[ \mathbb{P}(A)=\mathbb{P}(\cup_{n}B_{n})=\lim_{ n \to \infty }\sum_{i=1}^n\mathbb{P}(B_{i})=\lim_{ n \to \infty } \mathbb{P}(A_{n}). \] Then (5) holds similarly since decreasing \(\{ A_{n} \}\) gives increasing \(\{ A_{n}^c \}\) hence \[ \mathbb{P}(A)=1-\mathbb{P}(\cup_{n}A_{n}^c)=1-\lim_{ n \to \infty } \mathbb{P}(A_{n}^c)=\lim_{ n \to \infty } \mathbb{P}(A_{n}), \] completing the proof. \(\square\)
1.1.1 Random Variables
Random variables are mappings between measurable spaces, specifically between \((\Omega, \mathcal{F})\) and a measurable space defined on the real numbers \(\mathbb{R}\) for which we require a \(\sigma\)-algebra.
The Borel \(\sigma\)-algebra for some set \(X\), denoted \(\mathcal{B}_X\), is the generated \(\sigma\)-algebra for the topology of \(X\) i.e. the \(\sigma\)-algebra generated by the collection of open subsets of \(X\).
The members of the Borel \(\sigma\)-algebra \(\mathcal{B}_{X}\) are known as Borel sets.
The Borel \(\sigma\)-algebra for the real numbers \(\mathbb{R}\) is denoted \(\mathcal{R}=\mathcal{B}_{\mathbb{R}}\). Any measure that is defined on \((\mathbb{R},\mathcal{R})\) is known as a Borel measure and they are defined using a Stieltjes measure function \(F:\mathbb{R} \to \mathbb{R}\), a mapping that satisfies the following properties:
- Non-decreasing. \(F(x)\leq F(y)\) whenever \(x \leq y\).
- Right-continuous. \(\lim_{ y \downarrow x}F(y)=F(x)\) for every \(x \in \mathbb{R}\).
For each Stieltjes measure function \(F\) there exists a unique Borel measure \(\mu\) on the measure space \((\mathbb{R}, \mathcal{R})\) known as the Lebesgue-Stieltjes measure such that for all \(a<b\) \[ \mu((a,b])=F(b)-F(a). \] When \(F(x):=x\) for \(x \in \mathbb{R}\) this measure is specified as the Lebesgue measure and is denoted \(\lambda\).
A mapping \(X:\Omega \to \mathbb{R}\) between the underlying sets of two measurable spaces is said to be measurable if it preserves structure of the spaces, that is the preimage of any measurable set is measurable. This is in direct analogy to the definition that a continuous function between topological spaces preserves the topological structure: the preimage of any open set is open.
A random variable is a function \(X(\omega):\Omega \to \mathbb{R}\) from the (abstract) sample space \(\Omega\) to the real numbers \(\mathbb{R}\) such that for every Borel set \(B \in \mathcal{R}\) \[ X^{-1}(B):=\{ \omega \in \Omega :X(\omega) \in B \}\in \mathcal{F}, \] or equivalently stated, random variables are \(\mathcal{F}\)-measurable mappings between \((\Omega, \mathcal{F})\) and \((\mathbb{R}, \mathcal{R})\).
The \(\sigma\)-field generated by a random variable \(X\) is defined \[ \sigma(X):=\{ \{ \omega \in \Omega:X(\omega)\in B \},B \in \mathcal{R} \}, \] or intuitively stated, it is the collection of the pre-images of all Borel sets \(B\) with respect to \(X\).
A random variable \(X\) induces a (Borel) probability measure \(\mu\) on \((\mathbb{R},\mathcal{R})\) associated to \(\mathbb{P}\) known as the induced probability distribution of \(X\) and defined as \[ \mu(A)=\mathbb{P}(X(\omega )\in A)=\mathbb{P}(\omega:X(\omega )\in A)=\mathbb{P}(X^{-1}(A)). \]
Here we consider two probability spaces \((\Omega , \mathcal{F}, \mathbb{P})\) and \((\mathbb{R}, \mathcal{R}, \mu)\). We pull back the set \(A \in\mathcal{R}\) to the set \(X^{-1}(A) \in \mathcal{F}\) and associate the measure \(\mu\), sometimes writing \(\mu=\mathbb{P}\circ X^{-1}\). If two random variables \(X\) and \(Y\) induce the same distribution \(\mu\) on \((\mathbb{R}, \mathcal{R})\) we say that \(X\) and \(Y\) are equal in law (distribution) and write \[ X\stackrel{\mathcal{L}}{=}Y, \] and this holds if and only if \(X\) and \(Y\) have the same distribution function
For each \(x \in \mathbb{R}\) with \(A=(-\infty,x]\) we define a (cumulative) distribution function \[ F(x)=\mu(A)=\mathbb{P}(X \leq x). \]
Any distribution function \(F\) has the following properties:
- Monotonicity. \(F(x)\leq F(y)\) whenever \(x \leq y\).
- \(\lim_{ x \to \infty }F(x)=1\) and \(\lim_{ x \to -\infty }F(x)=0\)
- Right-continuity. \(\lim_{ y\downarrow x } F(y)=F(x)\).
- Left-limits. \(F(x-):=\lim_{ y\downarrow x }F(y)=\mathbb{P}(X<x)\).
- Jump size of \(F\) at \(x\). \(\mathbb{P}(X=x)=F(x)-F(x-)\).
If \(F\) is differentiable in \(\mathbb{R}\) and is of the form \[ F(x)=\int_{-\infty}^x f(y)dy, \] for \(x \in \mathbb{R}\) for some finite function \(f\), then we call \(f\) the probability density function and we say that \(F\) is absolutely continuous with respect to the Lebesgue measure.
1.1.2 Integration and Expectation
A measure \(\mu\) on measure space \((\Omega, \mathcal{F}, \mu)\) is said to be finite when \(\mu(\Omega)=1\) (hence all probability measures are finite). A measure is said to be \(\sigma\)-finite if the set \(\Omega\) can be covered with at most countably many measurable disjoint sets with finite measure. All finite measures are naturally also \(\sigma\)-finite.
We say that an event occurs almost surely (a.s.) if the (probability) measure of the set of outcomes where the event does not occur is \(0\).
On a measure space \((\Omega, \mathcal{F}, \mu)\) with \(\sigma\)-finite \(\mu\) we can define the Lebesgue integral of a measurable function \(f:\Omega \to \mathbb{R}\). Therefore, we can define a Lebesgue integral of a measurable random variable with respect to a probability space since it will have \(\sigma\)-finite measure and in fact, this is the exact definition of the expectation of a random variable.
For probability space \((\Omega, \mathcal{F}, \mathbb{P})\) we can define the expectation of a \(\mathcal{F}\)-measurable function \(f\) of random variable (measurable map) \(X:\Omega \to \mathbb{R}\) with respect to probability measure \(\mathbb{P}\) as the Lebesgue integral \[ \mathbb{E}f(X):=\int_{\Omega}f(X)d\mathbb{P}=\int_{\Omega}f(X(\omega))d\mathbb{P}(\omega). \]
This integral satisfies the following important properties:
- If \(X\geq 0~a.s.\) then \(\int_{\Omega}Xd\mathbb{P}\geq 0\),
- If \(a \in \mathbb{R}\) then \(\int_{\Omega}aXd\mathbb{P}=a\int_{\Omega}Xd\mathbb{P}\),
- \(\int_{\Omega}X+Y d\mathbb{P}=\int Xd\mathbb{P}+\int Yd\mathbb{P}\),
- If \(X \leq Y~a.s.\) then \(\int_{\Omega}Yd\mathbb{P}\geq \int_{\Omega}Xd\mathbb{P}\),
- If \(X=Y~a.s.\) then \(\int_{\Omega}Xd\mathbb{P}=\int_{\Omega}Yd\mathbb{P}\),
- \(\left\lvert \int_{\Omega}Xd\mathbb{P} \right\rvert\leq \int_{\Omega}\lvert X \rvert d\mathbb{P}\).
The proof of these properties comes from the construction of this integral and is omitted for efficiency. For more details see our note on the Lebesgue Integral.
For (\(\sigma\)-) finite \(\mathbb{P}\) with Stieltjes measure function \(G\) on the real line, i.e. \(\mu((a,b])=G(b)-G(a)\) we write \[ \int_{\Omega}f(\omega)d\mu(\omega) = \int_{\mathbb{R}}f(x)dG(x). \] Particularly, when \(\mu\) is the Lebesgue measure i.e. \(G(x)=x\) then we write \(\int_{\mathbb{R}}f(x)d(x)\). The following result describes how we can move between the traditional Riemann integral definition of expectation and the measure-theoretic definition.
For every \(S\)-valued measurable map \(X\) with distribution \(\mu(\cdot)=\mathbb{P}(X \in \cdot)=\mathbb{P}\circ X^{-1}(\cdot)\) and for \(g:(S,\mathcal{S})\to(\mathbb{R}, \mathcal{R})\) with \(g \geq 0\) or \(\mathbb{E}\lvert g(X) \rvert<\infty\) we have the change of variable formula \[ \mathbb{E}[g(X)]=\int_{\Omega}g(X(\omega))d\mathbb{P}(\omega)=\int_{S}g(y)d\mu(y). \]
1.1.3 Lp Spaces and Banach Spaces
Since random variables are functions we introduce the notion of \(\mathcal{L}^p\) spaces (sometimes known as Lebesgue spaces), a function categorization system which form an important class of [[banach-spaces]], by first defining the \(\mathcal{L}^p\)-norm.
For a random variable \(X\) in probability space \((\Omega , \mathcal{F}, \mathbb{P})\) the \(L^p\)-norm for \(1 \leq p\leq \infty\) is defined as \[ \lVert X \rVert_{p}:= \left( \mathbb{E}\lvert X \rvert^p \right)^{1/p}. \]
For fixed \(p \geq 1\) we denote by \(\mathcal{L}^p\) the collection of random variables \(X\) (measurable functions) such that \[ \lVert X \rVert_{p}= \int_{\Omega}\lvert X(\omega) \rvert ^p d\mathbb{P}(\omega)< \infty. \]
We say that all functions \(f\in \mathcal{L}^1\) are integrable.
Intuitively, we can think of functions as continuous vectors and by extension, function spaces are essentially infinite dimensional vector spaces. We can show that all \(\mathcal{L}^{p}\) spaces are complete, that is every Cauchy sequence \(\{ X_{n} \}\in \mathcal{L}_{p}\) converges to an element of \(\mathcal{L}_{p}\) in norm, i.e. \[ \lVert X_{n}-X_{m} \rVert_{p} \stackrel{n,m\to \infty}{\longrightarrow} 0 \iff \lVert X_{n}-X \rVert \stackrel{n \to \infty}\longrightarrow 0\quad\text{where}\quad X \in \mathcal{L}_{p}. \] We will talk more about convergence of sequences of random variables (stochastic processes) in a later section. In section [[]] we will also show that \(\mathcal{L}_{2}\)-spaces are a subclass of Banach spaces known as a Hilbert space which has a defined inner product.
1.1.4 Integral Properties
Markov’s inequality gives an upper bound on the probability that a non-negative random variable is greater than or equal to some positive constant. Markov’s inequality is tight in the sense that for each chosen positive constant, there exists a random variable such that the inequality is in fact an equality.
If \(X\) is a non-negative random variable, then for every \(\epsilon>0\) we have that \[ \mathbb{P}(X>\epsilon) \leq \frac{1}{\epsilon}\cdot \mathbb{E}X. \]
Proof: From the definition of expectation we have that \[ \begin{align}\mathbb{E}X & = \int _{-\infty}^\infty xf(x) \, dx \geq\int _{0}^\infty xf(x)\, dx =\int _{0}^\epsilon xf(x) \, dx +\int _{\epsilon}^\infty xf(x) \, dx \\ & \geq \int _{\epsilon}^\infty xf(x) \, dx \geq \int_{\epsilon}^\infty af(x) \, dx =\epsilon\mathbb{P}(X> \epsilon),\end{align} \] as required. \(\square\)
Jensen’s inequality gives a lower bound on expectations of convex functions including linear functions, quadratic functions, exponential functions and the modulus function.
For convex function \(\varphi\) and random variable \(X\) on a probability space \((\Omega, \mathcal{F}, \mathbb{P})\) we have \[ \varphi(\mathbb{E} X)\leq\mathbb{E}\varphi(X)~. \]
Proof: A convex function \(\varphi\) can be written as \(\varphi(x)=\sup_{a_{n}, b_{n}}(a_{n}x+b_{n})\), \(x \in \mathbb{R}\) for some \(\{ (a_{n}, b_{n}), n \geq 1 \}\subset \mathbb{R}^2\) of upper envelope for a countable family of affine functions. Then, taking the supremum of both sides of \[ \mathbb{E}\varphi(X)\geq \mathbb{E}[a_{n}X+b_{n}]=a_{n}\mathbb{E}[X]+b_{n};\quad n\geq 1, \] we obtain the result. \(\square\)
Let \(X\) and \(Y\) be random variables, \(p,q \in (1,\infty)\) and \(\frac{1}{p}+\frac{1}{q}=1\). We have that \[\int_{\Omega}^{}{\lvert X(\omega)Y(\omega) \rvert }~d{\mu(\omega)}=|\mathbb{E}[XY]|\leq \mathbb{E}|XY|\leq (\mathbb{E}|X|^p)^{1/p}\cdot (\mathbb{E}|Y|^q)^{1/q}=\lVert X \rVert_{p}\lVert Y \rVert _{q} .\]
Proof: For every \(a,b \geq 0\), \(\lambda \in (0,1)\) we have \[ a^\lambda b^{1-\lambda}\leq a\lambda + b(1-\lambda), \] where the equality holds with \(a=b\) (to verify evaluate \(\varphi(u)=u^\lambda-u\lambda\) over \([0,\infty)\)). When \(\lVert X \rVert_{p}>0\) and \(\lVert Y \rVert_{q}>0\) substituting \[ a=\left( \frac{\lvert X(\omega) \rvert}{\lVert X \rVert_{p} } \right)^p;\quad b=\left( \frac{\lvert Y(\omega) \rvert }{\lVert Y \rVert _{q}} \right) ^q;\quad \&\quad \lambda = \frac{1}{p}, \] into this inequality we obtain \[ \frac{\lvert X(\omega)Y(\omega) \rvert }{\lVert X \rVert _{p}\cdot \lVert Y \rVert _{p}}\leq \frac{\lvert X(\omega) \rvert ^p}{p\lVert X \rVert _{p}^p}+\frac{\lvert Y(\omega) \rvert ^q}{q\lVert Y \rVert _{q}^q}, \] for every \(\omega \in \Omega\) with equality when \(\lVert Y \rVert_{p}^p\cdot \lVert X(\omega) \rVert^p=\lVert X \rVert_{p}^p\cdot \lvert Y(\omega) \rvert^q\). Integrating both sides over \(\Omega\) with respect to \(\mu\) we obtain the result. \(\square\)
Both the Cauchy-Schwartz Inequality and Lyapunov's Inequality, given below, are
From Hölder's Inequality with values \(Y=1\) and \(p=2\) we obtain the Cauchy-Schwartz Inequality \[\mathbb{E}|X|\leq (\mathbb{E}|X|^2)^{1/2}.\]
From Hölder's Inequality replacing \(\lvert X \rvert\) by \(\lvert X \rvert^r\) with \(r>0\) and writing \(s=rp\) we obtain Lyapunov’s Inequality \[ (\mathbb{E}[\lvert X \rvert^r] )^{1/r}\leq (\mathbb{E}[\lvert X \rvert ^s])^{1/s};\quad 0 < r < s < \infty. \]
Let \(S\) be a measure space, let \(1\leq p\leq \infty\) and let \(f\) and \(g\) be elements of \(L^p(S)\). Then \(f+g\) is in \(L^p(S)\) and we have the Minkowski inequality \[\|f+g\|_{p}\leq \|f\|_{p}+\|g\|_{p}.\]
Assume that for some random variable \(X\) we have that \(\mathbb{E}X<\infty\). Then \[\mathbb{P}(|X-\mathbb{E}X|\geq \epsilon )\leq {\frac{\operatorname{Var}(X)}{\epsilon ^2}}.\]
1.2 2 Convergence of Random Variables
Random variables are functions. Consider a sequences of random variables \(\{X_{n}, n\geq 1\}\) and \(X\) on some probability space \((\Omega,\mathcal{F}, \mathbb{P})\). There are four important types of random variable convergence: (1) almost surely; (2) convergence in norm; (3) convergence in (probability) measure; and (4) convergence in law.
Note that although random variables (like any function) can converge pointwise this type of convergence is not helpful in probability theory.
A sequence of random variables \(\{X_n\}_{n\in\mathbb{N}}\) on probability space \((\Omega, \mathcal{F}, \mathbb{P})\) is said to converge almost surely to \(X\), denoted \(X_n\stackrel{a.s.}{\rightarrow}X\) \((n\rightarrow\infty)\) if \[ \mathbb{P}\left(\lim_{n\rightarrow\infty}X_n=X\right)=\mathbb{P}\left(\left\{\omega\in\Omega:\lim_{n\rightarrow\infty}X_n(\omega)=X(\omega)\right\}\right)=1. \]
A sequence of random variables \(\{ X_{n} \}_{n\in\mathbb{N}}\) convergence in \(L^p\)-norm to \(X\) denoted \(X_{n}\stackrel{L^p}{\to}X\) if \[\lim_{ n \to \infty }\Vert X_{n}-X\Vert_{p}=\lim_{ n \to \infty }(\mathbb{E}\left[|X_{n}-X|^p\right])^{1/p} = 0.\]
A sequence of random variables \(\{X_n\}_{n\in\mathbb{N}}\) on probability space \((\Omega, \mathcal{F}, \mathbb{P})\) converges in probability to \(X\), denoted \(X_n\stackrel{\mathbb{P}}{\rightarrow}X\) \((n\rightarrow\infty)\) if \[\lim_{ n \to \infty } \mathbb{P}(\{\omega \in\Omega:|X_{n}(\omega)-X(\omega)|>\epsilon\})=0\]for every \(\epsilon>0\).
A sequence of random variables \(\{X_n\}_{n\in\mathbb{N}}\) on probability space \((\Omega, \mathcal{F}, \mathbb{P})\) is said to converge in law (or converge in distribution) to \(X\) as \(n\to \infty\), denoted \(X_{n}\stackrel{\mathcal{D}}{\to}X\) if \[ \lim_{ n \to \infty } F_{X_{n}}(x)=F_{X}(x), \] that is, their sequence of [[distribution-function|distribution functions]] converge.
These types of convergence are not of equal strengths. The strongest types are almost surely and in norm, followed by convergence in measure and finally convergence in law. The diagram below summarizes the hierarchy of random variable convergence.
Figure: A graphical diagram of the random variable convergence hierarchy.
The continuous mapping theorem states that continuous functions preserve limits even if their arguments are sequences of random variables.
Let \(\{ X_{n} \}\) be a sequence of random variables and \(g\) some continuous function. Then - \(X_n\stackrel{\mathcal{D}}{\to}X_n\implies g(X_n)\stackrel{\mathcal{D}}{\to}g(X)\) - \(X_n\stackrel{\mathbb{P}}{\to}X_n\implies g(X_n)\stackrel{\mathbb{P}}{\to}g(X)\) - \(X_n\stackrel{a.s.}{\to}X_n\implies g(X_n)\stackrel{a.s.}{\to}g(X)\)
Proof:
The intuition behind the result is clear, since Heine’s definition of a continuous function is one that maps convergent sequences onto convergent sequences.
1.2.1 2.1 Convergence Almost Surely
1.2.2 2.2 The Borel-Cantelli Lemma
If \(X_{n}\stackrel{a.s}{\to}X\) as \(n \to \infty\) then \(X_{n}\stackrel{\mathbb{P}}{\to}X\) as \(n \to \infty\).
Proof: We fix \(\varepsilon>0\) and define the event \(A_{n}=\{ |X_{n}-X| \geq \epsilon \}\). We can then define the decreasing set \(B_{m}=\bigcup_{k=m}^\infty A_{k}\) i.e. \(B_{m}\supseteq B_{m+1}\supseteq\dots\) and since decreasing sequences have limits we have that \[B_{m}=\bigcup_{k=m}^\infty A_{k}\downarrow B_{\infty}=\bigcap_{m=0}^\infty \bigcup_{k=m}^\infty A_{k}=\limsup_{n\to \infty}A_{n}.\]We consider the limit \[\lim_{n \to \infty}\mathbb{P}(|X_{n}-X|>\varepsilon)=\lim_{n \to \infty}\mathbb{P}(A_{n})\leq \lim_{{n \to \infty}}\mathbb{P}(B_{n})\stackrel{m.c.t}{=}\mathbb{P}\left(\lim_{n\to \infty}B_{n}\right)\leq \mathbb{P}(X_{n}\not\to X)=0,\]completing the proof. \(\square\)
If \(X_{n}\stackrel{\mathbb{P}}{\to}X\), then there exists a non-random subsequence \(\{ n_{1}, n_{2}, \dots, n_{k}, \dots \}\nearrow \infty\) such that \(X_{n_{k}}\stackrel{a.s.}{\to}X\) as \(k \to \infty\).
Proof: Assuming \(X_{n}\stackrel{\mathbb{P}}{\to}X\) we fix integer \(k>0\) and choose \(\varepsilon = \frac{1}{k}\) such that by definition \[\lim_{n \to \infty}\mathbb{P}\left( \left\{ \omega \in \Omega:|X_{n}(\omega)-X(\omega)|> \frac{1}{k} \right\} \right) =0.\]Thus, there exists \(n_{k}\) such that \(\mathbb{P}\left( \left\{ \omega \in \Omega:|X_{n_{k}}(\omega)-X(\omega)|> \frac{1}{k} \right\} \right)=\mathbb{P}(A_{k})\leq \frac{1}{k}^2\). From results for Geometric Series we have that \[\sum_{i=1}^\infty\mathbb{P}(A_{k})=\sum_{i=1}^\infty \frac{1}{k^2}<\infty,\]and so from the Borel-Cantelli Lemma we have that \(\mathbb{P}(A_{k}~i.o)=0\) and so there exists \(n_{0}\) such that \(A_{k}^c=\left\{ \omega \in \Omega:\lvert X_{n_{k}}(\omega)-X(\omega) \rvert \leq \frac{1}{k} \right\}\) happens for all \(k \geq n_{0}\) with probability 1. \(\square\)
1.2.3 2.3 Convergence in Norm
We first show that higher orders of norm convergence implies lower orders.
Consider \(1<r<s<\infty\) and assume that \(X_{n}\stackrel{L^s}{\to}X\) as \(n \to \infty\). Then we have that \(X_{n}\stackrel{L^r}{\to}X\). The converse does not hold in general.
Proof: We apply Lyapunov's Inequality \[(\mathbb{E}|X|^r)^{1/r}\leq(\mathbb{E}|X|^s)^{1/s};\quad 0<r<s<\infty,\]to obtain the result. \(\square\)
Next we show that convergence in \(L_{1}\) norm implies convergence in probability, hence so to do all higher orders of convergence in norm from the previous result.
If \(X_{n}\stackrel{\mathcal{L}^1}{\to}X\) then \(X_{n}\stackrel{\mathbb{P}}{\to}X\) as \(n \to \infty\). The converse does not hold in general.
Proof: By applying Markov Inequality since, for every \(\epsilon>0\) we have that \[\mathbb{P}(|X_{n}-X|>\epsilon)\leq \frac{1}{\epsilon}\cdot \mathbb{E}|X_{n}-X|,\]giving the result. \(\square\)
For \(p>0\) assume that \(X_n \in \mathcal{L}^p\), \(n \geq 1\) and \(X_{n}\stackrel{\mathbb{P}}{\to} X\). The following three statements are equivalent: 1. \(\{ |X_{n}|^p,~n\geq 1 \}\) is uniformly integrable; 2. \(X_{n}\stackrel{L^p}{\to}X\) and \(X\in \mathcal{L}^p\); and 3. \(\lim_{n \to \infty}\mathbb{E}[|X_{n}|^p]=\mathbb{E}[|X|^p]<\infty\).
Proof:
1.2.4 2.4 Convergence in Probability
We have previously shown how convergence in probability follows from both convergence almost surely and convergence in norm. We have also shown how convergence in probability implies the existence of a subsequence that converges almost surely. It remains to show that convergence in probability implies convergence in law.
If \(X_{n}\stackrel{{\mathbb{P}}}{\to}X\) as \(n \to \infty\), then \(X_{n}\stackrel{\mathcal{D}}{\to}X\) as \(n\to \infty\).
Proof: For every \(x \in\mathbb{R}\), \(\epsilon>0\) we have that \[ \mathbb{P}(X\leq x-\epsilon)\leq \liminf_{n \to \infty } \mathbb{P}(X_{n}\leq x)\leq \limsup_{ n \to \infty }\mathbb{P}(X_{n}\leq x)\leq \mathbb{P}(X\leq x+\epsilon). \] If \(x\) is a continuity point of \(X\), then as \(\epsilon\downarrow 0\), the left and right sides converge and hence \[ \lim_{ n \to \infty } \mathbb{P}(X_{n}\leq x)=P(X\leq x), \] giving the result. \(\square\)
1.2.5 2.5 Convergence in Law
Unlike the other types, convergence in law tells us nothing about the behavior of the random variables themselves, only their distribution. Convergence in law means that \(F_{n}(x) \to F(x)\) for all \(x\) up to the points of discontinuity of \(F\). Limits in distribution are unique, that is \(F_{n}(x) \to F(x)\) and \(F_{n}(x) \to G(x)\) then \(F=G\).
The key convergence in law result is Skorohod's Representation Theorem, a result proving that a convergent in law sequence of probability measures whose limit measure is sufficiently well-behaved can be represented as the distribution / law of a pointwise convergent sequence of random variables defined on a common probability space.
Suppose that \(X_{n}\stackrel{\mathcal{D}}{\to}X\) as \(n \to \infty\) with \(F_{n}(x):=\mathbb{P}(X_{n} \leq x)\) and \(F(x):=\mathbb{P}(X\leq x)\) for \(x \in \mathbb{R}\). Then, there exists a probability space \((\Omega', \mathcal{F}',\mathbb{P}')\) and random variables \(\{Y_{n}, n\geq 1\}\) and \(Y\) such that \[ Y_{n}\stackrel{\mathcal{D}}{=}X_{n},\quad Y\stackrel{\mathcal{D}}{=X}, \] for every \(n\) and \(Y_{n}\stackrel{a.s.}{\to}Y\) as \(n \to \infty\).
Proof:
There are several equivalent definitions of weak convergence of a sequence of measures, some of which are (apparently) more general than others. The equivalence of these conditions is sometimes known as the Portmanteau Theorem.
The following statements are equivalent: 1. \(X_{n}\stackrel{\mathcal{D}}{\to}X\); 2. \(\mathbb{E}[g(X_{n})]\to \mathbb{E}[g(X)]\) for all bounded and continuous functions \(g\); 3. \(\mathbb{E}[g(X_{n})]\to \mathbb{E}[g(X)]\) for all \(g\) of the form \(g(x)=f(x)\mathbb{1}_{[a,b]}(x)\), where \(f\) is continuous on \([a,b]\) and \(a,b \in C(F)\), that is, \(g\) is a continuous function with finite support \([a,b]\) with \(a,b \in C(F)\); 4. \(\mathbb{E}[g(X_{n})]\to \mathbb{E}[g(X)]\) for all \(g\) bounded and uniformly continuous; 5. If \(A\) is such that \(\mathbb{P}(X \in \delta A)=0\) then \(\mathbb{P}(X_{n}\in A)\to \mathbb{P}(X\in A).\)
In (5) \(\delta A\) denotes the boundary of a set \(A\). These are points that are reachable both form inside the set and outside: \[ \delta A=\{ x : \exists \{ y_{n} \}\in A~s.t. y_{n}\to x~\&~ \exists \{ z_{n} \}\in A^c~s.t.~z_{n} \to x \}. \] Equivalent formulation of (5) using d.f.s \(F_{n}\) and \(F\): \[ F(\delta A)=0\implies F_{n}(A)\to F(A). \] Here \(F((a,b]):=F(b)-F(a)=\int \mathbb{1}_{(a,b]}(x)dF(x)\).
Proof:
In this rest of this section we consider some more detailed questions about the convergence of sequences of distribution functions. We will be begin by showing that a sequence of distribution functions don’t necessarily converge using Helly's Selection Theorem to provide a partial answer.
Helly’s Selection Theorem is the first of several results (including tightness and its resulting theorems) forming a sort of compactness result for certain families of probability measures. The theorem essentially states that any sequence of distribution functions contains a subsequence that converges to a right-continuous non-decreasing function.
Any sequence of distribution functions \(\{ F_{n} \}\) contains a convergent subsequence \(\{ F_{n} \}\) contains a convergent subsequence \(\{ F_{n_{k}} \}: F_{n_{k}}(x) \to F(x)\) as \(k \to \infty\) for some right-continuous non-decreasing function \(F\). Convergence holds \(\forall x \in C(F)\).
The limit \(F\) of a sequence of distribution function \(\{ F_{n} \}\) is itself a distribution function when the sequence \(\{ F_{n} \}\) is tight. This is helpful as we then have that any sequence of random variables \(\{ X_{n} \}\) whose distribution functions \(\{ F_{n} \}\) contain a subsequence \(\{ X_{n_{k}} \}\) who converge weakly.
Fatou’s Lemma states that if \(X_{n}\geq 0\) then \[ \liminf_{n \to \infty}\mathbb{E}[X_{n}] \geq \mathbb{E}\left[\liminf_{n \to \infty}X_{n}\right]. \]
Monotone Convergence Theorem: Let \((X_{n})_{n \in \mathbb{N}}\) be a sequence of non-negative monotonically increasing random variables on \((\Omega, \mathcal{F}, \mathbb{P})\) where \(X_{n}\stackrel{a.s}\to X\). Then we have that \[ \lim_{n\to \infty}\mathbb{E}[X_{n}]=\mathbb{E}\left[\lim_{n \to \infty}X_{n}\right]=\mathbb{E}[X]. \] Dominated Convergence Theorem: For sequence of random variables \(\{ X_{n} \}\) such that the pointwise limit \(X(\omega)=\lim_{n\to \infty}X_{n}(\omega)\) exists, assume there is an integrable random variable \(Y\) with \(\lvert X(\omega) \rvert\leq Y(\omega)\) for all \(\omega \in \Omega\), then \(X\) is integrable as in \(X_n\) for all \(n\) and \[ \lim_{n \to \infty}\mathbb{E}[X_{n}]=\mathbb{E}\left[\lim_{n\to \infty} X_{n}\right]=\mathbb{E}[X]. \]
1.3 3 Laws of Large Numbers
1.3.1 3.1 Independence
Collections of sets \(\mathcal{A}_{1}, .., \mathcal{A}_{n}\subset \mathcal{F}\) are independent if whenever \(A_{i} \in \mathcal{A}_{i}\) and \(I \subset \{ 1, \dots, n \}\) we have \[ \mathbb{P}\left( \bigcap _{i=1}^n A_{i} \right) =\prod_{i=1}^{n}\mathbb{P}(A_{i}). \] If each collection is a single set \(\mathcal{A}_{i}=\{ A_{i} \}\) then this definition reduces to the one for sets.
Without loss of generality suppose each \(\mathcal{A}_i\) contains \(\Omega\). Then the condtion for independence is equivalent to \[ \mathbb{P}\left( \bigcap _{i=1}^n A_{i} \right) =\prod_{i=1}^{n}\mathbb{P}(A_{i})~~\text{whenever}~~ A_{i}\in \mathcal{A}_{i}, \] since we can set \(A_{i}=\Omega\) for \(i \not\in I\).
Proof: If \(\mathcal{A}_{1}, \dots, \mathcal{A}_{n}\) are independent and \(\overline{\mathcal{A}}_{i}=\mathcal{A}_{i}\cup \{ \Omega \}\) then \(\overline{A}_{1}, \dots, \overline{\mathcal{A}}_{n}\) are independent since if \(A_{i} \in \overline{\mathcal{A}}_{i}\) and \(I = \{ j: A_{j} = \Omega \}\) then \(\bigcap_{i}A_{i}=\bigcap_{i \in I}A_{i}\). \(\square\)
To prove the key result of this section we are going to require the Dynkin's π-λ Theorem otherwise known as the \(\pi-\lambda\) theorem.
1.3.2 3.2 Dynkin’s π-λ Theorem
References: [[probability-theory-and-examples-durrett-pdf|Durrett §2.1.1]], NEU Online Note.
1.3.3 3.3 Weak Law of Large Numbers
References: [[probability-theory-and-examples-durrett-pdf|Durrett]]
1.3.4 3.4 Strong Law of Large Numbers
Strong Law of Large Numbers ### 3.5 Characteristic Functions
1.4 4 Limit Theorems
1.4.1 4.1 Central Limit Theorem
1.4.2 4.2 Limit Theorems
1.5 5 Stochastic Processes
1.5.1 5.1 Conditional Expectation
From [[]] we discussed how all \(\mathcal{L}_{p}\) spaces are Banach spaces, that is they are complete linear vector spaces with norms given by the \(\mathcal{L}_{p}\) norm \[ \lVert X \rVert_{p}:= \left( \mathbb{E}\lvert X \rvert^p \right)^{1/p}. \] We specifically the consider space \(\mathcal{L}_{2}(\Omega , \mathcal{F}, \mathbb{P})\) of random variables on \((\Omega, \mathcal{F}, \mathbb{P})\) with finite second moments \(\mathbb{E}[X^2]<\infty\). We can show this is a subclass of Banach spaces known as a Hilbert space by defining an inner product \(\left< \cdot, \cdot \right>:\Omega \times \Omega \to \mathbb{R}\) by \[ \left< X,Y \right> := \mathbb{E}[XY], \] from which we obtain the \(L_{2}\) norm \(\lVert X \rVert_{2} = \sqrt{ \mathbb{E}[X^2] }\).
Any subset \(\mathcal{F}_{1}\subseteq \mathcal{F}\) that is also a \(\sigma\)-algebra is known as a sub-\(\sigma\)-algebra. Then the function space \(\mathcal{L}_{2}^*:=\mathcal{L}_{2}(\Omega,\mathcal{F}_{1}, \mathbb{P})\) is a closed subspace of \(\mathcal{L}_{2}\). For an arbitrary element \(X \in \mathcal{L}_{2}\), the distance from \(X\) to subspace \(\mathcal{L}_{2}^*\) is given by \[ d:=\inf_{Y\in \mathcal{L}_{2}^*}\left\{ \lVert X-Y \rVert _{2} \right\}. \] The function \(Y^*\in\mathcal{L}_{2}^*\) that minimizes this distance, i.e. \(d=\lVert X-Y^* \rVert\) is said to be the orthogonal projection \(X\) onto \(\mathcal{L}_{2}^*\). Therefore we have that \[ \lVert X - Y^* \rVert_{2} \leq \lVert X - Y \rVert_{2} ,~~\forall~ Y\in \mathcal{L}_{2}^*. \] The [[existence-theorem|existence theorem]] ensures that such a function exists for all Hilbert spaces. Furthermore, the [[projection-theorem|projection theorem]] then gives us a specific way to explicitly compute the orthogonal projection.
Let \(X\in\mathcal{L}_{2}\) and \(Y\in\mathcal{L}_{2}^*\). Then the following statements are equivalent \[ \lVert X - Y^* \rVert_{2} \leq \lVert X - Y \rVert_{2} ,~~\forall Y \in\mathcal{L}_{2}^*~~\equiv~~\left< X-Y^*, Y \right> = 0,~~\forall Y \in\mathcal{L}_{2}^*. \]
Proof: See Grimmett TB Section 7.9 pg 390.
We interpret \(\sigma\)-algebra as information and so \(\mathbb{E}[X|\mathcal{F}]\) can be thought of as our best estimate of the random variable \(X\) given the available information \(\mathcal{F}\). The concept of our ‘best estimate’ is equivalent to orthogonality (minimizing distance i.e. \(L_{2}\) norm) giving the following definition.
For random variable \(X\in \mathcal{L}_{2}(\Omega, \mathcal{F}, \mathbb{P})\) and sub-\(\sigma\)-algebra \(\mathcal{F}_{1}\subseteq \mathcal{F}\) the conditional expectation \(\mathbb{E}[X|\mathcal{F}_{1}]\) is defined as the orthogonal projection of \(X\) onto \(\mathcal{L}_{2}^* :=\mathcal{L}_{2}(\Omega,\mathcal{F}_{1}, \mathbb{P})\), that is, a random variable that satisfies \[\left< X-\mathbb{E}[X|\mathcal{F}_{1}], Y \right> = \mathbb{E}[(X-\mathbb{E}[X|\mathcal{F}_{1}])\cdot Y]=0~~\forall Y \in \mathcal{L}_{2}^* .\]
In the next theorem we outline several of the most important properties of conditional expectation.
For random variables \(X,Y \in\mathcal{L}_{2}(\Omega, \mathcal{F}, \mathbb{P})\) the following results hold almost surely: 1. \(\mathbb{E}[\alpha X+\beta_{Y}|\mathcal{F}_{1}]=\alpha\mathbb{E}[X|\mathcal{F}_{1}]+\beta\mathbb{E}[Y|\mathcal{F}_{1}]\) (linearity); 2. \(X=C\) a.s. \(\implies \mathbb{E}[X|\mathcal{F}_{1}]=C\) a.s.; 3.
1.5.2 5.2 Markov Processes
1.5.2.1 5.2.1 Generating Functions
1.5.2.2 5.2.2 Poisson Processes
[[poisson-processes]] ### 5.3 Martingales
1.5.3 5.4 Stochastic Processes
1.5.4 5.5 Brownian Motion
- Markov Chains
- Stochastic Processes
- Brownian Motion