1 Probability Spaces
A probability space is a triple \((\Omega, \mathcal{F}, \mathbb{P})\) consisting of a sample space \(\Omega\), a σ-algebra \(\mathcal{F}\) and a probability measure \(\mathbb{P}:\mathcal{F}\to[0,1]\).
The sample space is the abstract collection of possible outcomes of a random experiment. The σ-algebra is the collection of all groups of outcomes called events. The probability measure is a function describing the probability of these events.
A random experiment is any repeatable procedure or process with a well-defined set of possible outcomes, where the exact result cannot be predicted in advance. A sample space \(\Omega\) is the collection of outcomes \(\omega \in \Omega\) of a random experiment.
The sample space for a single roll of a 6-sided die is
\[ \Omega :=\{ 1,2,3,4,5,6 \}. \]
Subsets of \(\Omega\) (i.e. collections of outcomes) are known as events and are often denoted by capitalized letters (e.g. \(A\)).
We can define the event of rolling an event number by \(A:=\{ 2,4,6 \}.\)
In analysis, an [[algebra]] is a collection of sets that is non-empty, and closed under finite unions and compliments. A \(\sigma\)-algebra is a specific type of algebra that is closed under countable unions.
A \(\sigma\)-algebra \(\mathcal{F}\) for a sample set \(\Omega\) is a collection of subsets of \(\Omega\) (i.e. collection of events) that satisfies the following properties:
- Non-empty: \(\Omega \in\mathcal{F}\).
- Closed under compliments: \(A\in\mathcal{F} \implies A^c\in\mathcal{F}\).
- Closed under countable unions:\(A_{i}\in\mathcal{F}\) for countable \(i\) \(\implies \cup_{i}A_{i}\in\mathcal{F}\).
The σ-algebra for our dice rolling experiment is the power set
\[ \mathcal{F}:=2^\Omega = \{\emptyset, \{1\}, \{2\}, \ldots, \{1,2,3,4,5,6\}\}, \]
which contains all 64 subsets of \(\Omega\).
Therefore a \(\sigma\)-algebra is simply an [[Algebra|algebra]] that is closed under countable unions. Some intuitive examples include:
- \(\mathcal{F}=2^\Omega\) (the power set of the sample space)
- \(\mathcal{F}=\{ \emptyset, \Omega \}\) (the trivial \(\sigma\)-algebra)
- \(\Omega=\{ 1,2,3 \}\implies \mathcal{F}= \{ \emptyset, \{ 1 \}, \{ 2,3 \}, \Omega \}\).
For a (possibly uncountable) collection of \(\sigma\)-algebras \(\mathcal{F}_{i}\) with index set \(i\in I\) over sample space \(\Omega\) the intersection \(\cap_{i}\mathcal{F}_{i}\) is also a \(\sigma\)-algebra.
Proof. We have that:
- \(\forall i,\Omega \in\mathcal{F}_{i} \implies \Omega \in\cap_{i}\mathcal{F}_{i}\).
- Let \(A\in\cap_{i}\mathcal{F}_{i}\implies \forall i, A\in\mathcal{F}_{i}\implies \forall i,A^c\in\mathcal{F}_{i}\implies A^c\in\cap_{i}\mathcal{F}_{i}\).
- Let \(\{ A_{n} \}_{n\in\mathbb{N}}\subseteq \cap_{i}\mathcal{F}_{i}\)
For a collection \(\mathcal{A}\subseteq 2^\Omega\) of subsets of \(\Omega\) there is a unique smallest \(\sigma\)-algebra on \(\Omega\) containing \(\mathcal{A}\), known as the \(\sigma\)-algebra generated by \(\mathcal{A}\) defined as
\[ \sigma(\mathcal{A})=\bigcap \{ \mathcal{F}:\mathcal{F}\text{ is a }\sigma \text{-algebra over }\Omega~s.t.~\mathcal{A}\subseteq \mathcal{F} \}. \]
From the study of [[Measure Theory]], a [[Measure]] is a generalized function measuring the size of a set.
A probability measure \(\mathbb{P}\) on measurable space \((\Omega,\mathcal{F})\) is a mapping \(\mathbb{P}:\mathcal{F}\mapsto [0,1]\) satisfying the following properties:
- \(\mathbb{P}(A)\geq \mathbb{P}(\emptyset)=0\);
- \(\mathbb{P}(\Omega)=1\);
- For countable disjoint \(A_{i}\in\mathcal{F}\) then \(\mathbb{P}(\cup_{i}A_{i})=\sum_{i}\mathbb{P}(A_{i})\).
For measure \(\mathbb{P}\) on probability space \(\mathbb{P}\) we have the following basic properties:
- Monotonicity: \(A\subset B,~A,B\in \mathcal{F}\implies \mathbb{P}(A) \leq \mathbb{P}(B)\);
- Rule of Addition: \(A,B \in \mathcal{F}\implies \mathbb{P}(A \cup B)+ \mathbb{P}(A\cap B)=\mathbb{P}(A)+\mathbb{P}(B)\);
- Subadditivity: \(\{ A_{n} \}_{n\in\mathbb{N}}\subseteq \mathcal{F}\implies P(\cup_{n=1}^\infty A_{n})\leq \sum_{n=1}^\infty \mathbb{P}(A_{n})\);
- Continuity from below: \(A_{i}\uparrow A\) (i.e \(A_1 \subset A_2 \subset\dots~\&~\bigcup_{i}A_{i}=A\)) \(\implies \mathbb{P}(A_{i})\uparrow \mathbb{P}(A)\).
- Continuity from above: \(A_{i}\downarrow A\implies \mathbb{P}(A_{i}) \downarrow \mathbb{P}(A)\).
Proof.
- Monotonicity: Since \(A \subset B\), we have \(B = A \cup (B \setminus A)\) (disjoint), so \(\mathbb{P}(B) = \mathbb{P}(A) + \mathbb{P}(B \setminus A) \geq \mathbb{P}(A)\) as \(\mathbb{P}(B \setminus A) \geq 0\).
- Rule of Addition: Write \(A \cup B = A \cup (B \setminus A)\) (disjoint) and \(B = (A \cap B) \cup (B \setminus A)\) (disjoint). Then \(\mathbb{P}(A \cup B) = \mathbb{P}(A) + \mathbb{P}(B \setminus A)\) and \(\mathbb{P}(B) = \mathbb{P}(A \cap B) + \mathbb{P}(B \setminus A)\). Thus \(\mathbb{P}(A \cup B) = \mathbb{P}(A) + \mathbb{P}(B) - \mathbb{P}(A \cap B)\).
- Subadditivity: Define \(B_1 = A_1, B_n = A_n \setminus (A_1 \cup \cdots \cup A_{n-1})\) for \(n \geq 2\). These are disjoint with \(\bigcup B_n = \bigcup A_n\). By countable additivity, \(\mathbb{P}(\bigcup A_n) = \sum \mathbb{P}(B_n)\). Since \(B_n \subseteq A_n\), \(\mathbb{P}(B_n) \leq \mathbb{P}(A_n)\), so \(\mathbb{P}(\bigcup A_n) \leq \sum \mathbb{P}(A_n)\).
- Continuity from below: Write \(A = A_1 \cup (A_2 \setminus A_1) \cup (A_3 \setminus A_2) \cup \cdots\) (disjoint). By countable additivity, \(\mathbb{P}(A) = \mathbb{P}(A_1) + \sum_{i=1}^{\infty} \mathbb{P}(A_{i+1} \setminus A_i)\). Since \(\mathbb{P}(A_n) = \mathbb{P}(A_1) + \sum_{i=1}^{n-1} \mathbb{P}(A_{i+1} \setminus A_i)\), we have \(\mathbb{P}(A_n) \to \mathbb{P}(A)\) as \(n \to \infty\).
- Continuity from above: Write \(A_1 \setminus A = (A_1 \setminus A_2) \cup (A_2 \setminus A_3) \cup \cdots\) (disjoint). By countable additivity, \(\mathbb{P}(A_1) - \mathbb{P}(A) = \sum_{i=1}^{\infty} \mathbb{P}(A_i \setminus A_{i+1})\). Note that \(\mathbb{P}(A_n) - \mathbb{P}(A) = \sum_{i=n}^{\infty} \mathbb{P}(A_i \setminus A_{i+1}) \to 0\) as \(n \to \infty\).
2 Borel Sets
The Borel σ-algebra is a σ-algebra defined on the real numbers \(\mathbb{R}\).
Taking \(\Omega:=\mathbb{R}\), the smallest σ-Algebra that contains \(\mathbb{R}\) is called the Borel σ-algebra \(\mathcal{R}=\mathcal{B}_{\mathbb{R}}: =\sigma(\mathbb{R})\). Any set in \(\mathcal{R}\) we call a Borel set.
More rigorously, for a topological space \(\mathcal{T}\), the Borel σ-algebra of \(\mathcal{T}\) is the σ-algebra generated from the open sets of \(\mathcal{T}\).
That is, it is the smallest collection of subsets of \(\mathcal{T}\) that contain all the open sets of \(\mathcal{T}\) and satisfy the properties of a σ-algebra.
Any measure that is defined on \(\mathcal{R}\) is called a Borel measure.
3 Lebesgue Measure
To define measures on the real numbers \(\mathbb{R}\) we first define Stieltjes measure functions which are non-decreasing, right continuous real functions.
On the measurable space \((\mathbb{R},\mathcal{R})\) a Stieltjes measure function \(F:\mathbb{R} \to \mathbb{R}\) is any function that is:
- Non-decreasing. \(F(b) \geq F(a)\) for all \(b > a\).
- Right continuous. \(\lim_{y \downarrow x}F(y) = F(x)\).
For each Stieltjes measure function we can define a unique measure called the Stieltjes measure.
For each Stieltjes measure function \(F\) there exists a unique Borel measure \(\mu\) on the measure space \((\mathbb{R}, \mathcal{R})\) known as the Lebesgue-Stieltjes measure satisfying
\[ \mu((a,b])=F(b)-F(a), \]
for every \(a<b\).
When \(F(x):=x\) for \(x \in \mathbb{R}\) this measure is specified as the Lebesgue measure and is specifically denoted by \(\lambda\).
Proof.
First we define a [[Algebra|semi-algebra]] \(\mathcal{S}\), an algebra \(\bar{\mathcal{S}}\), and then a (\(\sigma\)-finite) measure \(\mu\) on the algebra \(\bar{\mathcal{S}}\).
We extend a set function \(\mu\) on \(\mathcal{S}\) to a measure \(\bar{\mathcal{\mu}}\) on \(\bar{\mathcal{S}}\) under the conditions of (1) finite additivity and (2) sub-additivity. We check (1) finite additivity for every union of disjoint sets and (2) sub-additivity for every union of disjoint sets.
We extend a \(\sigma\)-finite measure \(\bar{\mu}\) on algebra \(\bar{S}\) uniquely to a measure \(\nu\) on \(\sigma(S)\) via the [[caratheodory-extension-theorem]].