0.1 Transformations of Random Variables
References: Arizona State Note.
Suppose we have some random variable \(X\) with a given density function \(f_{X}\). Further consider a real valued function \(g:\mathbb{R} \to \mathbb{R}\) and define the new random variable \(Y=g(X)\). This note is on the methods available for determining the properties of this new random variable, such as the expectation, the moments or the distribution.
Computing the expectation of \(Y=g(X)\) given the density of \(X\) is simple using the following result sometimes known as the fundamental theorem.
Given random variable \(X\) with density \(f_{X}\) and real-valued function \(g:\mathbb{R} \to \mathbb{R}\) the expectation of transformed random variable \(Y = g(X)\) is given by \[ \mathbb{E}[Y] = \int_{\Omega}^{}{g(X)}~d{\mathbb{P}} =\int_{X}^{}{g(X)\cdot f_{X}(x)}~d{x}. \]
The discrete case clearly becomes a sum on the discrete support of \(X\) with mass function \(p_{X}(x)\).
We note that in general \(\mathbb{E}[g(X)]\neq g(\mathbb{E}[X])\). In fact, the only time we can guarantee equality is when \(g\) is an Affine Function, that is can be written as a combination of a linear function and a translation.
Recall from our study of functional analysis, the inverse image of real function \(g: \mathbb{R} \to \mathbb{R}\) on a set \(A\) is \[ g^{-1}(A) := \{ x \in \mathbb{R}:g^{-1}(x) \in A \}. \] This is the same as saying that \(x \in g^{-1}(A)\) if and only if \(g(x) \in A\). The inverse image has the following properties:
- \(g^{-1}(\mathbb{R})=\mathbb{R}\);
- For any set \(A\), \(g^{-1}(A^c)=g^{-1}(A)^c\); and
- For any collection of sets \(\{ A_{\lambda}:\lambda \in \Lambda \}\) we have \(g^{-1}(\cup_{\lambda}A_{\lambda})=\cup_{\lambda}g^{-1}(A)\).
As a result, the mapping \(\mu(A)=\mathbb{P}(X \in g^{-1}(A))\) satisfies the axioms of probability and is known as the distribution of \(X\) (see notes on Probability Theory).
0.1.1 Discrete Transformations
Assume that the random variable \(X\) is discrete with probability mass function \(f_{X}(x)\). To compute the probability mass function \(f_{Y}(y)\) of random variable \(Y = g(X)\) we simply use \[ f_{Y}(y)=\sum_{x \in g^{-1}(y)}^{}f_{X}(x). \] Example: Let \(X\sim\mathcal{P}(\lambda)\) follow a Poisson Distribution with rate parameter \(\lambda\) and define the [[polynomial-function|quadratic function]] \(Y = g(X)= X^2+7\). The support of \(X\) is \(\mathbb{N}_{0}\) and on this support \(g\) is one-to-one and so the mass function \(f_{Y}\) is given by \[ f_{Y}(y)=f_{X}(g^{-1}(y))=f_{X}\left( \sqrt{ y-7 }\right)=\frac{\lambda^{\sqrt{ y-7 }}e^{-\lambda}}{\left(\sqrt{ y-7 }\right)!}, \] on the support \(y=\{x^2+7:x \in \mathbb{N}\}=7, 8, 11, \dots\).
0.1.2 Continuous Transformations
The simplest case of transformations of continuous random variables is when the transformation function \(g\) is one-to-one, i.e. a function is either strictly increasing or strictly decreasing. In both cases the inverse function \(g^{-1}\) is also either strictly increasing or strictly decreasing respectively. We can then employ what is known as the CDF method where we can compute the CDF of \(Y = g(X)\) as \[ \mathbb{P}(Y \leq y)=\mathbb{P}(g(X)\leq y)=\mathbb{P}(X \leq g^{-1}(y))=F_{X}(g^{-1}(y)). \] From this we can compute the density using the chain run \[ f_{Y}(y)=f_{X}(g^{-1}(y))\cdot \left| \frac{d}{dy}g^{-1}(y) \right|, \] where we take the modulus to combine both the increasing and decreasing cases.
Special care needs to be taken when the transform \(g\) is not one-to-one, for example when \(g(x)=x^2\) or \(g(x) =|x|\) and the support of \(x\) is \(\mathbb{R}\).
For example, when \(g(x) = x^2\) we have \(g^{-1}(x)=\sqrt{ x }\) and so we have that \[ F_{Y}(y)=\mathbb{P}(Y \leq y) = \mathbb{P}(X^2\leq y)=\mathbb{P}(-\sqrt{y} \leq X \leq \sqrt{ y } )=F_{X}(\sqrt{ y })-F_{X}(-\sqrt{ y }). \] The density is then given as \[ \begin{align} f_{Y}(y) & =f_{X}(\sqrt{ y }) \frac{d}{dy}\sqrt{ y }-f_{X}(-\sqrt{ y }) \frac{d}{dy}(-\sqrt{ y }) \\ & = \frac{1}{2 \sqrt{ y }}\left\{ f_{X}(\sqrt{ y })+f_{X}(\sqrt{ y }) \right\} . \end{align} \] Further, if the density \(f_{X}\) is symmetric about the origin then we have \[ f_{Y}(y)= \frac{1}{ \sqrt{ y }}f_{X}(\sqrt{ y }). \] Example: Consider a random variable \(Z\) following the standard normal distribution with density \[ \phi(z)= \frac{1}{\sqrt{ 2\pi }}\exp\left( -\frac{z^2}{2} \right). \] We wish to compute the density of \(Y = Z^2\). First we note that \[ \phi'(z)= \frac{1}{\sqrt{ 2\pi }}\exp\left( -\frac{z^2}{2} \right)\cdot -z=-z\phi(z). \] The density is therefore \[ f_{Y}(y)=\frac{1}{\sqrt{ 2\pi y }}\exp\left( - \frac{y}{2} \right), \] which is a Chi-Squared Distribution.
0.1.3 The Probability Transform
Let \(X\) be a continuous random variable whose distribution function \(F_{X}\) is strictly increasing on the support of \(X\). Then \(F_{X}\) is invertible. Letting \(U=F_{X}(X)\), then for \(u \in [0,1]\) \[ \mathbb{P}(U \leq u) = \mathbb{P}(F_{X}(X)\leq u) = \mathbb{P}(U \leq F_{X}^{-1}(u))=F_{X}(F_{X}^{-1}(u))=u. \] In other words, \(U\) is a uniform random variable on \([0,1]\). Most random number generators simulate independent copies of this random variable. Consequently, we can simulate independent random variables having distribution function \(F_{X}\) by simulating \(U\), a uniform random variable on \([0,1]\) and then taking \[ X= F_{X}^{-1}(U). \]