Probability Spaces

What is a probability space?

---

A probability space is a triple $(\Omega ,\mathcal{F},\mathbb{P})$ consisting of a sample space $\Omega$, a σ-algebra $\mathcal{F}$ and a probability measure $\mathbb{P}:\mathcal{F}\to [0,1]$.

The sample space is the abstract collection of possible outcomes of a random experiment. The σ-algebra is the collection of all groups of outcomes called events. The probability measure is a function describing the probability of these events.

A random experiment is any repeatable procedure or process with a well-defined set of possible outcomes, where the exact result cannot be predicted in advance. A sample space $\Omega$ is the collection of outcomes $\omega \in \Omega$ of a random experiment.

Example: The sample space for a single roll of a 6-sided die is $\Omega :=\{1,2,3,4,5,6\}.$

Subsets of $\Omega$ (i.e. collections of outcomes) are known as events and are often denoted by capitalized letters (e.g. $A$).

Example: We can define the event of rolling an event number by $A:=\{2,4,6\}.$

What is a σ-Algebra?

---

In analysis, an algebra is a collection of sets that is non-empty, and closed under finite unions and compliments.

Definition: σ-Algebra

A $\sigma$-algebra $\mathcal{F}$ for a sample set $\Omega$ is a collection of subsets of $\Omega$ (i.e. collection of events) that satisfies the following properties:

1. Non-empty: $\Omega \in \mathcal{F}$.
2. Closed under compliments: $A\in \mathcal{F}⟹A^c\in \mathcal{F}$.
3. Closed under countable unions:$A_i\in \mathcal{F}$ for countable $i$ $⟹{\cup }_i{A}_i\in \mathcal{F}$.

Example: The σ-algebra for our dice rolling experiment is the power set $\mathcal{F}:=2^{\Omega }=\{\varnothing ,\{1\},\{2\},\dots ,\{1,2,3,4,5,6\}\},$ which contains all 64 subsets of $\Omega$.

Therefore a $\sigma$-algebra is simply an algebra that is closed under countable unions. Some intuitive examples include:

1. $\mathcal{F}=2^{\Omega }$ (the power set of the sample space)
2. $\mathcal{F}=\{\varnothing ,\Omega \}$ (the trivial $\sigma$-algebra)
3. $\Omega =\{1,2,3\}⟹\mathcal{F}=\{\varnothing ,\{1\},\{2,3\},\Omega \}$.

Theorem: Intersection of $\sigma$-Algebra

For a (possibly uncountable) collection of $\sigma$-algebras ${\mathcal{F}}_i$ with index set $i\in I$ over sample space $\Omega$ the intersection ${\cap }_i{\mathcal{F}}_i$ is also a $\sigma$-algebra.

Proof: We have that:

1. $\forall i,\Omega \in {\mathcal{F}}_i⟹\Omega \in {\cap }_i{\mathcal{F}}_i$.
2. Let $A\in {\cap }_i{\mathcal{F}}_i⟹\forall i,A\in {\mathcal{F}}_i⟹\forall i,A^c\in {\mathcal{F}}_i⟹A^c\in {\cap }_i{\mathcal{F}}_i$.
3. Let $\{A_n{\}}_{n\in \mathbb{N}}\subseteq {\cap }_i{\mathcal{F}}_i$
   $\square$

Theorem: Generated σ-Algebra

For a collection $\mathcal{A}\subseteq 2^{\Omega }$ of subsets of $\Omega$ there is a unique smallest $\sigma$-algebra on $\Omega$ containing $\mathcal{A}$, known as the $\sigma$-algebra generated by $\mathcal{A}$ defined as $\sigma (\mathcal{A})=\bigcap \{\mathcal{F}:\mathcal{F}\text{ is a }\sigma \text{-algebra over }\Omega s.t.\mathcal{A}\subseteq \mathcal{F}\}.$

What is a Probability Measure?

---

From the study of Measure Theory, a Measure is a generalized function measuring the size of a set.

Definition: Probability Measure

A probability measure $\mathbb{P}$ on measurable space $(\Omega ,\mathcal{F})$ is a mapping $\mathbb{P}:\mathcal{F}\mapsto [0,1]$ satisfying the following properties:

1. $\mathbb{P}(A)\ge \mathbb{P}(\varnothing )=0$;
2. $\mathbb{P}(\Omega )=1$;
3. For countable disjoint $A_i\in \mathcal{F}$ then $\mathbb{P}({\cup }_i{A}_i)={\sum }_i\mathbb{P}(A_i)$.

What are the key properties of a probability measure?

---

For measure $\mathbb{P}$ on probability space $\mathbb{P}$ we have the following basic properties:

1. Monotonicity: $A\subset B,A,B\in \mathcal{F}⟹\mathbb{P}(A)\le \mathbb{P}(B)$;
2. Rule of Addition: $A,B\in \mathcal{F}⟹\mathbb{P}(A\cup B)+\mathbb{P}(A\cap B)=\mathbb{P}(A)+\mathbb{P}(B)$;
3. Subadditivity: $\{A_n{\}}_{n\in \mathbb{N}}\subseteq \mathcal{F}⟹P({\cup }_{n=1}^{\infty }{A}_n)\le {\sum }_{n=1}^{\infty }\mathbb{P}(A_n)$;
4. Continuity from below: $A_i\uparrow A$ (i.e $A_1\subset A_2\subset \dots \&{\bigcup }_i{A}_i=A$) $⟹\mathbb{P}(A_i)\uparrow \mathbb{P}(A)$.
5. Continuity from above: $A_i\downarrow A⟹\mathbb{P}(A_i)\downarrow \mathbb{P}(A)$.

Proof:

1. Monotonicity: Since $A\subset B$, we have $B=A\cup (B\setminus A)$ (disjoint), so $\mathbb{P}(B)=\mathbb{P}(A)+\mathbb{P}(B\setminus A)\ge \mathbb{P}(A)$ as $\mathbb{P}(B\setminus A)\ge 0$.
2. Rule of Addition: Write $A\cup B=A\cup (B\setminus A)$ (disjoint) and $B=(A\cap B)\cup (B\setminus A)$ (disjoint). Then $\mathbb{P}(A\cup B)=\mathbb{P}(A)+\mathbb{P}(B\setminus A)$ and $\mathbb{P}(B)=\mathbb{P}(A\cap B)+\mathbb{P}(B\setminus A)$. Thus $\mathbb{P}(A\cup B)=\mathbb{P}(A)+\mathbb{P}(B)-\mathbb{P}(A\cap B)$.
3. Subadditivity: Define $B_1=A_1,B_n=A_n\setminus (A_1\cup \cdots \cup A_{n-1})$ for $n\ge 2$. These are disjoint with $\bigcup B_n=\bigcup A_n$. By countable additivity, $\mathbb{P}(\bigcup A_n)=\sum \mathbb{P}(B_n)$. Since $B_n\subseteq A_n$, $\mathbb{P}(B_n)\le \mathbb{P}(A_n)$, so $\mathbb{P}(\bigcup A_n)\le \sum \mathbb{P}(A_n)$.
4. Continuity from below: Write $A=A_1\cup (A_2\setminus A_1)\cup (A_3\setminus A_2)\cup \cdots$ (disjoint). By countable additivity, $\mathbb{P}(A)=\mathbb{P}(A_1)+{\sum }_{i=1}^{\infty }\mathbb{P}(A_{i+1}\setminus A_i)$. Since $\mathbb{P}(A_n)=\mathbb{P}(A_1)+{\sum }_{i=1}^{n-1}\mathbb{P}(A_{i+1}\setminus A_i)$, we have $\mathbb{P}(A_n)\to \mathbb{P}(A)$ as $n\to \infty$.
5. Continuity from above: Write $A_1\setminus A=(A_1\setminus A_2)\cup (A_2\setminus A_3)\cup \cdots$ (disjoint). By countable additivity, $\mathbb{P}(A_1)-\mathbb{P}(A)={\sum }_{i=1}^{\infty }\mathbb{P}(A_i\setminus A_{i+1})$. Note that $\mathbb{P}(A_n)-\mathbb{P}(A)={\sum }_{i=n}^{\infty }\mathbb{P}(A_i\setminus A_{i+1})\to 0$ as $n\to \infty$.

What is a Borel σ-Algebra?

---

The Borel σ-algebra is a σ-algebra defined on the real numbers $\mathbb{R}$.

Definition: Borel σ-Algebra

Taking $\Omega :=\mathbb{R}$, the smallest σ-Algebra that contains $\mathbb{R}$ is called the Borel σ-algebra $\mathcal{R}={\mathcal{B}}_{\mathbb{R}}:=\sigma (\mathbb{R})$. Any set in $\mathcal{R}$ we call a Borel set.

More rigorously, for a topological space $\mathcal{T}$, the Borel σ-algebra of $\mathcal{T}$ is the σ-algebra generated from the open sets of $\mathcal{T}$. That is, it is the smallest collection of subsets of $\mathcal{T}$ that contain all the open sets of $\mathcal{T}$ and satisfy the properties of a σ-algebra.

Any measure that is defined on $\mathcal{R}$ is called a Borel measure.

What is a Lebesgue Measure?

---

To define measures on the real numbers $\mathbb{R}$ we first define Stieltjes measure functions which are non-decreasing, right continuous real functions.

Definition: Stieltjes Measure Function

On the measurable space $(\mathbb{R},\mathcal{R})$ a Stieltjes measure function $F:\mathbb{R}\to \mathbb{R}$ is any function that is:

1. Non-decreasing. $F(b)\ge F(a)$ for all $b>a$.
2. Right continuous. ${\lim }_{y\downarrow x}F(y)=F(x)$.

For each Stieltjes measure function we can define a unique measure called the Stieltjes measure.

Theorem: Lebesgue-Stieltjes Measure

For each Stieltjes measure function $F$ there exists a unique Borel measure $\mu$ on the measure space $(\mathbb{R},\mathcal{R})$ known as the Lebesgue-Stieltjes measure satisfying $\mu ((a,b])=F(b)-F(a),$for every $a<b$.

When $F(x):=x$ for $x\in \mathbb{R}$ this measure is specified as the Lebesgue measure and is specifically denoted by $\lambda$.

Proof:

1. First we define a semi-algebra $\mathcal{S}$, an algebra $\bar{\mathcal{S}}$, and then a ($\sigma$-finite) measure $\mu$ on the algebra $\bar{\mathcal{S}}$.
2. We extend a set function $\mu$ on $\mathcal{S}$ to a measure $\bar{\mu }$ on $\bar{\mathcal{S}}$ under the conditions of (1) finite additivity and (2) sub-additivity. We check (1) finite additivity for every union of disjoint sets and (2) sub-additivity for every union of disjoint sets.
3. We extend a $\sigma$-finite measure $\bar{\mu }$ on algebra $\bar{S}$ uniquely to a measure $\nu$ on $\sigma (S)$ via the Carathéodory Extension Theorem.