1 Introduction
Suppose we have some random variable \(X\) with a given density function \(f_{X}\). Further consider a real valued function \(g:\mathbb{R} \to \mathbb{R}\) and define the new random variable \(Y=g(X)\). This note is on the methods available for determining the properties of this new random variable, such as the expectation, the moments or the distribution.
2 Fundamental Theorem
Computing the expectation of \(Y=g(X)\) given the density of \(X\) is simple using the following result sometimes known as the fundamental theorem.
Given random variable \(X\) with density \(f_{X}\) and real-valued function \(g:\mathbb{R} \to \mathbb{R}\) the expectation of transformed random variable \(Y = g(X)\) is given by
\[ \mathbb{E}[Y] = \int_{\Omega}^{}{g(X)}~d{\mathbb{P}} = \int_{X}^{}{g(X)\cdot f_{X}(x)}~d{x}. \]
The discrete case clearly becomes a sum on the discrete support of \(X\) with mass function \(p_{X}(x)\).
We note that in general \(\mathbb{E}[g(X)]\neq g(\mathbb{E}[X])\). In fact, the only time we can guarantee equality is when \(g\) is an affine function, that is can be written as a combination of a linear function and a translation.
Recall from our study of Real Analysis, the inverse image of real function \(g: \mathbb{R} \to \mathbb{R}\) on a set \(A\) is
\[ g^{-1}(A) := \{ x \in \mathbb{R}:g^{-1}(x) \in A \}. \]
This is the same as saying that \(x \in g^{-1}(A)\) if and only if \(g(x) \in A\). The inverse image has the following properties:
- \(g^{-1}(\mathbb{R})=\mathbb{R}\);
- For any set \(A\), \(g^{-1}(A^c)=g^{-1}(A)^c\); and
- For any collection of sets \(\{ A_{\lambda}:\lambda \in \Lambda \}\) we have \(g^{-1}(\cup_{\lambda}A_{\lambda})=\cup_{\lambda}g^{-1}(A)\).
As a result, the mapping \(\mu(A)=\mathbb{P}(X \in g^{-1}(A))\) satisfies the axioms of probability and is known as the distribution of \(X\) (see notes on Probability Theory).
3 Discrete Transformations
Assume that the random variable \(X\) is discrete with probability mass function \(f_{X}(x)\). To compute the probability mass function \(f_{Y}(y)\) of random variable \(Y = g(X)\) we simply use
\[ f_{Y}(y)=\sum_{x \in g^{-1}(y)}^{}f_{X}(x). \]
Let \(X\sim\mathcal{P}(\lambda)\) follow a [[poisson-distribution]] with rate parameter \(\lambda\) and define the quadratic function \(Y = g(X)= X^2+7\). The support of \(X\) is \(\mathbb{N}_{0}\) and on this support \(g\) is one-to-one and so the mass function \(f_{Y}\) is given by
\[ f_{Y}(y)=f_{X}(g^{-1}(y))=f_{X}\left( \sqrt{ y-7 }\right) = \frac{\lambda^{\sqrt{ y-7 }}e^{-\lambda}}{\left(\sqrt{ y-7 }\right)!}, \]
on the support \(y=\{x^2+7:x \in \mathbb{N}\}=7, 8, 11, \dots\).
4 Continuous Transformations
The simplest case of transformations of continuous random variables is when the transformation function \(g\) is one-to-one, i.e. a function is either strictly increasing or strictly decreasing. In both cases the inverse function \(g^{-1}\) is also either strictly increasing or strictly decreasing respectively. We can then employ what is known as the CDF method where we can compute the CDF of \(Y = g(X)\) as
\[ \mathbb{P}(Y \leq y)=\mathbb{P}(g(X)\leq y) = \mathbb{P}(X \leq g^{-1}(y))=F_{X}(g^{-1}(y)). \]
From this we can compute the density using the chain run
\[ f_{Y}(y)=f_{X}(g^{-1}(y))\cdot \left| \frac{d}{dy}g^{-1}(y) \right|, \]
where we take the modulus to combine both the increasing and decreasing cases.
Special care needs to be taken when the transform \(g\) is not one-to-one, for example when \(g(x)=x^2\) or \(g(x) =|x|\) and the support of \(x\) is \(\mathbb{R}\).
For example, when \(g(x) = x^2\) we have \(g^{-1}(x)=\sqrt{ x }\) and so we have that
\[ F_{Y}(y)=\mathbb{P}(Y \leq y) = \mathbb{P}(X^2\leq y) =\mathbb{P}(-\sqrt{y} \leq X \leq \sqrt{ y } ) = F_{X}(\sqrt{ y })-F_{X}(-\sqrt{ y }). \]
The density is then given as
\[ \begin{align} f_{Y}(y) & = f_{X}(\sqrt{ y }) \frac{d}{dy}\sqrt{ y } - f_{X}(-\sqrt{ y }) \frac{d}{dy}(-\sqrt{ y }) \\ & = \frac{1}{2 \sqrt{ y }} \left\{ f_{X}(\sqrt{ y })+f_{X}(\sqrt{ y }) \right\} . \end{align} \]
Further, if the density \(f_{X}\) is symmetric about the origin then we have
\[ f_{Y}(y)= \frac{1}{ \sqrt{ y }}f_{X}(\sqrt{ y }). \]
Consider a random variable \(Z\) following the standard [[normal-distribution]] with density
\[ \phi(z)= \frac{1}{\sqrt{ 2\pi }}\exp\left( -\frac{z^2}{2} \right). \]
We wish to compute the density of \(Y = Z^2\). First we note that
\[ \phi'(z)= \frac{1}{\sqrt{ 2\pi }}\exp\left( -\frac{z^2}{2} \right)\cdot -z = -z\phi(z). \]
The density is therefore
\[ f_{Y}(y)=\frac{1}{\sqrt{ 2\pi y }}\exp\left( - \frac{y}{2} \right), \]
which is a [[chi-squared-distribution]].
5 The Probability Transform
Let \(X\) be a continuous random variable whose distribution function \(F_{X}\) is strictly increasing on the support of \(X\). Then \(F_{X}\) is invertible.
Letting \(U=F_{X}(X)\), then for \(u \in [0,1]\)
\[ \mathbb{P}(U \leq u) = \mathbb{P}(F_{X}(X)\leq u) = \mathbb{P}(U \leq F_{X}^{-1}(u))=F_{X}(F_{X}^{-1}(u))=u. \]
In other words, \(U\) is a uniform random variable on \([0,1]\). Most random number generators simulate independent copies of this random variable.
Consequently, we can simulate independent random variables having distribution function \(F_{X}\) by simulating \(U\), a uniform random variable on \([0,1]\) and then taking
\[ X= F_{X}^{-1}(U). \]