Random Variables and Distributions
Published in Fall, 2024
Random Variables
Definition. Let $(\Omega,\mathcal{F})$ be a sample space, suppose $X$ is a finite, singled valued function that maps $\Omega$ into $\mathbb{R}$, then $X$ is a random variable if $$ \forall B \in \mathfrak{B}_{\mathbb{R}} : X^{-1}(B) := \{ \omega \in X : X(\omega) \in B \} \in \mathcal{F} $$
That is, $X$ is a random variable if the inverse image under $X$ of all Borel sets in $\mathbb{R}$ are events in $\mathcal{F}$. As we mentioned earlier, we don’t need to check all Borel sets in $\mathbb{R}$ (e.g open sets, closed sets, half open half closed sets), we can just check some specific sets.
Theorem. $X$ is a random variable if and only if $\forall x \in \mathbb{R}$, we have $$ \{ \omega : X(\omega) \leq x \} \in \mathcal{F} $$
Proof. It is trivial because we know that $\mathfrak{B}_{\mathbb{R}} := \sigma( (a,b])$.
Corollary. If $X$ is a random variable, then $\forall a,b,c \in \mathbb{R}$, the following sets: $$ \{ X = c \} ; \{ a < X < b \} ; \{ a \leq X \leq b \} ; \{ a \leq X < b \} ;\{ X < c \};\cdots $$ are all events in $\mathcal{F}$.
Proof. Again, it is trivial by the construction of the Borel $\sigma$ field on $\mathbb{R}$.
Theorem. If $X$ is a random variable, then so is $Y = aX + b$, $a,b \in \mathbb{R}$.
Proof. For any given $x$, consider the set $$ \{ \omega : aX(\omega) + b \leq x \} $$ i.e $$ \{ \omega : aX(\omega) \leq x - b \} $$ $\bullet$ If $a>0$, then we have $\{ X(\omega) \leq \displaystyle{\frac{x-b}{a}} \} \in \mathcal{F}$;
$\bullet$ If $a < 0$, then we have $\{ X(\omega) \geq \displaystyle{\frac{x-b}{a}} \} \in \mathcal{F}$;
$\bullet$ If $a = 0$, then then $\{ \omega : aX(\omega) \leq x - b \} = \begin{cases} \{ \Omega \} : x - b \geq 0 \\ \varnothing : x - b < 0 \end{cases} \in \mathcal{F}$.
Thus $Y =aX+b$ is also a random variable.
$\bullet$ If $a < 0$, then we have $\{ X(\omega) \geq \displaystyle{\frac{x-b}{a}} \} \in \mathcal{F}$;
$\bullet$ If $a = 0$, then then $\{ \omega : aX(\omega) \leq x - b \} = \begin{cases} \{ \Omega \} : x - b \geq 0 \\ \varnothing : x - b < 0 \end{cases} \in \mathcal{F}$.
Thus $Y =aX+b$ is also a random variable.
Here are some examples about random variables:
- Example 1: For any set $A \in \mathcal{F}$, define the indicator function of set $A$ by
If we consider the pre-image given by $\mathbb{I}_A$, we will find
\[\mathbb{I}_A^{-1} ( (-\infty,x]) = \begin{cases} \varnothing : x < 0 \\ A^C : 0 \leq x < 1 \\ \Omega : x \geq 1 \end{cases}\]Since $\varnothing, \Omega \in \mathcal{F}$, so $\mathbb{I}_A$ is a random variable if and only if $A \in \mathbb{F}$.
- Example 2: Consider tossing a fair coin two times, denote $H$ = head face up and $T$ = tail face up, then we know that $\Omega = { HH, HT, TH, TT }$ and $\mathcal{F}$ is the $\sigma$ field generated by $\Omega$, we define $X(\omega) = \text{Number of $H$’s in $\omega$}$, then clearly $X(HH) = 2, X(HT) = X(TH) = 1, X(TT) = 0$, and further we have
and thus $X$ is a random variable.
Corollary. If $X,Y$ are random variables, then so is $X+Y$.
Probability Distribution of Random Variable
Definition. Let $(\Omega, \mathcal{F})$ be a sample space with a probability $\mathbb{P}$, then the space $(\Omega, \mathcal{F}, \mathbb{P})$ is called a probability space.
Theorem. The random variable $X$ defined on the probability space $(\Omega, \mathcal{F},\mathbb{P})$ can be mapped to another probability space $(\mathbb{R}, \mathfrak{B}{\mathbb{R}}, Q)$ by another probability function $Q$ defined by: $$ \forall B \in \mathfrak{B}_{\mathbb{R}} : Q(B) = \mathbb{P}\{ X^{-1}(B) \} = \mathbb{P} \{ \omega : X(\omega) \in B \} $$ We call $Q = \mathbb{P}(X^{-1})$ the probability distribution of $X$.
Proof. To prove that $Q$ is a probability, we just need to verify all 3 axioms from definition $7$. Clearly $Q(B) \geq 0, \forall B \in \mathfrak{B}_{\mathbb{R}}$, and $Q(\mathbb{R}) = \mathbb{P}(X \in \mathbb{R}) = \mathbb{P}(\Omega) = 1$, also $$ \begin{align*} Q \left( \sum_{i=1}^{\infty} B_i \right) = \mathbb{P} \{ X^{-1} \left( \sum_{i=1}^{\infty} B_i \right) \} &= \mathbb{P} \{ \sum_{i=1}^{\infty} X^{-1}(B_i) \} \\ & = \sum_{i=1}^{\infty} \mathbb{P}(X^{-1}(B_i)) = \sum_{i=1}^{\infty} Q(B_i) \end{align*} $$
Definition. A real valued function $F$ defined on $\mathbb{R}$ that is non-decreasing, right continuous and satisfies $$ F(-\infty) = 0, F(+\infty = 1) $$ is called a distribution function.
Definition. Let $X$ be a random variable defined on $(\Omega,\mathcal{F},\mathbb{P})$, define a point function $F$ on $\mathbb{R}$ by: $$ \forall x \in \mathbb{R} : F(x) = Q((-\infty,x]) = \mathbb{P}\{ \omega : X(\omega) \leq x \} $$ We call $F$ the distribution function of the random variable $X$.
It is not hard to verify that $F$ defined as above is indeed a distribution function.
Theorem. Given a probability $Q$ on $(\mathbb{R}, \mathfrak{B}_{\mathbb{R}})$, there exists a distinct distribution function $F$ such that $$ \forall x \in \mathbb{R} : Q((-\infty,x]) = F(x) $$
Now, we will study two types of random variable and their distribution function.
Definition. A random variable $X$ defined on $(\Omega,\mathcal{F},\mathbb{P})$ is said to be of the discrete type, if there exists a countable set $M \subset \mathbb{R}$ such that $\mathbb{P}(X \in M) = 1$, and the points of $M$ which have positive values are called jump points.
We know that all the singletons are Borel sets, thus ${ X \in E}$ is an event.
Definition. The collection of numbers $\{ p_i \}$ satisfying $\mathbb{P}\{ X = x_i \} = p_i \geq 0$ for all $i$ and $\sum_i p_i = 1$ is called the probability mass function (PMF) of the random variable $X$, and the distribution function of $X$ is given by $$ F(x) = \mathbb{P} \{ X \leq x \} = \sum_{x_i \leq x} p_i $$
We can use the indicator function $\mathbb{I}$ to further rewrite the definition, we now have
\[X(\omega) = \sum_{i} x_i \mathbb{I}_{X = x_i} (\omega)\]If we define
\[\epsilon(x) = \begin{cases} 1 : x \geq 0 \\ 0 : x < 0 \end{cases}\]Then
\[F(x) = \sum_i p_i \epsilon(x - x_i)\] Theorem. Let $\{ p_k \}$ be a collection of non-negative real numbers such that $\sum_{k} p_k = 1$, then $\{ p_k \}$ is the probability mass function (PMF) for some random variable $X$.
The other type of random variable is those without jump points.
Definition. Let $X$ be a random variable defined on $(\Omega, \mathcal{F}, \mathbb{P})$ with distribution function $F$, then $X$ is said to be of the continuous type if $F$ is absolutely continuous, i.e there exists a non-negative function $f(x)$ such that $$ \forall x \in \mathbb{R} : F(x) = \int_{-\infty}^x f(t) dt $$ The function $f$ is called the probability density function (PDF) of the random variable $X$. If $F$ is absolutely continuous and $f$ is continuous at $x$, then $$ F'(x) = \frac{dF(x)}{dx} = f(x) $$
Theorem
Let $X$ be a random variable of the continuous type with probability density function $f$, then $\forall B \in \mathfrak{B}{\mathbb{R}}$, we have \(\mathbb{P}(B) = \int_{B} f(t) dt\)
Theorem. Every non-negative real function $f$ that is integrable over $\mathbb{R}$ and satisfies $\displaystyle{\int_{-\infty}^{+\infty} f(x) dx = 1}$ is the probability density function for some random variable $X$
Proof. We need to find such a $F$, where $$ F(x) = \int_{-\infty}^x f(t)dt $$ Then by definition of $f(x)$, clearly $F(-\infty) = 0$, $F(+\infty) = 1$, and if $x_2 \geq x_1$, then $$ F(x_2) = \int_{-\infty}^{x_1} f(t) dt + \int_{x_1}^{x_2} f(t) dt \geq \int_{-\infty}^{x_1} f(t)dt = F(x_1) $$ So $F$ is non-decreasing. Furthermore $F$ is continuous (hence right-continuous), thus that finishes the proof.
Theorem. Let $X$ be a random variable, then $$ \mathbb{P}(x = a) := \lim_{t \to a^-} \mathbb{P}\{ t < x \leq a \} $$
Now let’s look at some examples:
Example. Suppose a random variable $X$ is distributed according to $f_X(x)$, written as $X \sim f_X(x)$, satisfies $$ X \sim f_X(x) = \begin{cases} x : 0<x\leq 1 \\ 2 - x : 1 < x \leq 2 \\ 0 : \text{otherwise} \end{cases} $$ Find the distribution function $F$.
Click to show/hide solution
Solution.
First we know that $\displaystyle{\int_{\mathbb{R}} f_X(x) dx = 1}$. By definition, $F(x)$ is given by $$ F(x) = \int_{-\infty}^x f(t) dt = \begin{cases} 0 : x \leq 0 \\ \\ \displaystyle{\int_0^x t dt = \frac{1}{2}x^2 : 0 < x \leq 1} \\ \\ \displaystyle{\int_{0}^1 tdt + \int_1^x (2-t)dt = 2x - \frac{1}{2}x^2 - 1 : 1 < x \leq 2 : 1 < x \leq 2} \\ \\ 1 : x > 2 \end{cases} $$
First we know that $\displaystyle{\int_{\mathbb{R}} f_X(x) dx = 1}$. By definition, $F(x)$ is given by $$ F(x) = \int_{-\infty}^x f(t) dt = \begin{cases} 0 : x \leq 0 \\ \\ \displaystyle{\int_0^x t dt = \frac{1}{2}x^2 : 0 < x \leq 1} \\ \\ \displaystyle{\int_{0}^1 tdt + \int_1^x (2-t)dt = 2x - \frac{1}{2}x^2 - 1 : 1 < x \leq 2 : 1 < x \leq 2} \\ \\ 1 : x > 2 \end{cases} $$
Example. Suppose $X$ is a random variable whose distribution function is given by $$ F(x) = \begin{cases} 0 : x <0 \\ \frac{1}{2} : x = 0 \\ \frac{1}{2} + \frac{1}{2}x : 0<x<1 \\ 1 : x \geq 1 \end{cases} $$ Then we notice that there is a jump point at $x=0$, so we may not differentiate directly, instead we write $F(x)$ as $$ F(x) = \frac{1}{2}F_d(x) + \frac{1}{2}F_c(x) $$ where $$ F_d(x) = \begin{cases} 0 : x<0 \\ 1 : x \geq 0 \end{cases} ; F_c(x) = \begin{cases} 0 : x \leq 0 \\ x : 0 < x < 1 \\ 1 : 1 \leq x \end{cases} $$ So in this case, $F_c(x)$ has no jump points, thus $$ f_c(x) = \frac{dF_c(x)}{dx} = \begin{cases} 1 : 0<x<1 \\ 0 :\text{otherwise} \end{cases} $$ and $F_d(x)$ is the distribution function of $X$ degenerate at $x=0$, i.e $F_d(x) = \mathbb{P}(x = 0) = 1$.
Functions of Random Variables
Theorem. Suppose $X$ is a random variable, let $g$ be a Borel measurable function on $\mathbb{R}$, then $Y = g(X)$ is also a random variable.
Proof. We let $Y = g(X)$, then $F_Y(y) := \{ g(X) \leq y \} = \{ X \in g^{-1}((-\infty,y] )\}$, which is measurable.
Example. Suppose $X$ is a random variable with distribution function $F_X$, then what about $Y = \vert X \vert$?
Click to show/hide solution
Solution.
We know that $$ F_Y(y) = \mathbb{P}(Y \leq y) = \mathbb{P}( \vert x \vert \leq y) = \mathbb{P}(-y \leq x \leq y) $$ If $X$ is of the continuous type, then we have $$ F_Y(y) = F_X(y) - F_X(-y), $$ if $X$ is of the discrete type, then we have $$ F_Y(y) = F_X(y) - F_X(-y^-), \text{since $X$ has a jump discontinuity at $-y$}. $$
We know that $$ F_Y(y) = \mathbb{P}(Y \leq y) = \mathbb{P}( \vert x \vert \leq y) = \mathbb{P}(-y \leq x \leq y) $$ If $X$ is of the continuous type, then we have $$ F_Y(y) = F_X(y) - F_X(-y), $$ if $X$ is of the discrete type, then we have $$ F_Y(y) = F_X(y) - F_X(-y^-), \text{since $X$ has a jump discontinuity at $-y$}. $$
Example. Suppose $X \sim N(0,1)$ where $$ f_X(x) = \frac{1}{\sqrt{2\pi }} e^{-\frac{x^2}{2}} $$ Where $X$ satisfies normal distribution, which general form is $$ X \sim N(\mu,\sigma^2), f_X(x) = \frac{1}{\sqrt{2\pi } \sigma} e^{-\frac{(x-\mu)^2}{2\sigma^2}}, $$ then what is $F_Y$, where $Y =\vert X \vert$?
Click to show/hide solution
Solution.
As from above, we know that $$ F_Y(y) = F_X(y) - F_X(-y) = \int_{-\infty}^y \frac{1}{\sqrt{2\pi }} e^{-\frac{x^2}{2}} dx - \int_{-\infty}^{-y} \frac{1}{\sqrt{2\pi }} e^{-\frac{x^2}{2}} dx. $$
As from above, we know that $$ F_Y(y) = F_X(y) - F_X(-y) = \int_{-\infty}^y \frac{1}{\sqrt{2\pi }} e^{-\frac{x^2}{2}} dx - \int_{-\infty}^{-y} \frac{1}{\sqrt{2\pi }} e^{-\frac{x^2}{2}} dx. $$
Theorem. If $X$ is a continuous random variable and $g$ is differentiable, then $g(X)$ is also a continuous random variable.
Back to the previous, we then have $$ f_Y(y) = \frac{dF_Y(y)}{dy} = \frac{dF_X(y)}{dy} - \frac{dF_X(-y)}{dy} = f_x(y) + f_X(-y), $$ thus $$ f_Y(y) = \frac{1}{\sqrt{2\pi }} e^{-\frac{y^2}{2}} + \frac{1}{\sqrt{2\pi }} e^{-\frac{(-y)^2}{2}} = \sqrt{\frac{2}{\pi }} e^{-\frac{y^2}{2}}. $$ Example. Suppose $X \sim Poisson(\lambda)$, given by $\displaystyle{\mathbb{P}(X = k) := \frac{e^{-\lambda} \lambda^k}{k!}}$ where $k \in \mathbb{N}_0, \lambda >0$. Now suppose $Y = X^2 + 3$, then $$ \mathbb{P}(Y =y) = \mathbb{P}(g(x) = y) = \mathbb{P}(x = \sqrt{y-3}) = \frac{e^{-\lambda} \lambda^{\sqrt{y-3}}}{(\sqrt{y-3})!}. $$
Theorem. Let $X$ be a random variable of the continuous type, with probability density function $f$. Let $g(x)$ be differentiable, and $\vert g'(x) \vert >0$ for all $x$, then $Y = g(X)$ is also a random variable of the continuous type with probability density function given by $$ h_Y(y) = \begin{cases} \displaystyle{f_X(g^{-1}(y)) \cdot \Bigg\vert \frac{d}{dy} g^{-1}(y) \Bigg\vert : \alpha < y < \beta} \\ \\ 0 : \text{otherwise} \end{cases} $$ where $\alpha = \min \{ g(-\infty), g(+\infty) \} ; \beta = \max\{ g(-\infty), g(+\infty) \}$.
Proof. Suppose $g'(x) >0$, then $g$ is continuous and strictly increasing, thus $\lim \alpha, \beta$ exists (possibly infinity), and its inverse $x = g^{-1}(y)$ also exists, differentiable, continuous and strictly increasing. The distribution function of $Y$ for $\alpha < y < \beta$ is given by $$ \mathbb{P}(Y \leq y) = \mathbb{P}(X \leq g^{-1}(y)) $$ The probability density function of $g$ is obtained by differentiation: $$ \begin{align*} h_Y(y) = \frac{d}{dy} F_Y(y) &= \frac{d}{dy} \mathbb{P}(Y \leq y) \\ &= \frac{d}{dy} \mathbb{P}(X \leq g^{-1}(y)) \\ &= \frac{d}{dy} F_X(g^{-1}(y)) \\ &= f_X(g^{-1}(y)) \cdot \frac{d}{dy} g^{-1}(y) \end{align*} $$ Likewise, when $g'(x)<0$, we have $$ h(y) =- f_X(g^{-1}(y)) \cdot \frac{d}{dy} g^{-1}(y) $$ And that completes the proof.
Example. Given that $$ X \sim f_X(x) = \begin{cases} \displaystyle{\frac{2x}{\pi^2} : 0 < x < \pi } \\ \\0 : \text{otherwise} \end{cases} $$ Let $Y = \sin(X)$, then find the probability density function of $Y$.
Click to show/hide solution
Solution.
Let $Y =\sin(X) = g(X)$, we have $\mathbb{P}(Y \leq y) = \mathbb{P}(\sin(x) \leq y) = \mathbb{P}(x \in (0,x_1) \cap x \in (x_2,\pi ) )$ where $x_1 = \sin^{-1}(y) , x_2 = \pi - \sin^{-1}(y)$. Thus we have $$ \begin{align*} \mathbb{P}(Y \leq y) &= \mathbb{P}(x \in (0, x_1)) + \mathbb{P}(x\in (x_2,\pi )) \\ & = \int_0^{x_1} f_X(x) dx + \int_{x_2}^{\pi } f_X(x) dx \\ & = \left( \frac{x_1}{\pi } \right)^2 + 1 - \left( \frac{x_2}{\pi } \right)^2 \end{align*} $$ So we have $$ h_Y(y) = \frac{d}{dy} F_Y(y) = \frac{d}{dy} \mathbb{P}(Y \leq y) = \frac{d}{dy} \left[ \left( \frac{\sin^{-1}(x_1)}{\pi } \right)^2 + 1 - \left( \frac{\sin^{-1}(x_2)}{\pi } \right)^2 \right] $$ and which simplifies to $$ h_Y(y) = \begin{cases} \displaystyle{\frac{2}{\pi \sqrt{1-y^2}}} : 0<y<1 \\ \\0 : \text{otherwise} \end{cases}. $$
Let $Y =\sin(X) = g(X)$, we have $\mathbb{P}(Y \leq y) = \mathbb{P}(\sin(x) \leq y) = \mathbb{P}(x \in (0,x_1) \cap x \in (x_2,\pi ) )$ where $x_1 = \sin^{-1}(y) , x_2 = \pi - \sin^{-1}(y)$. Thus we have $$ \begin{align*} \mathbb{P}(Y \leq y) &= \mathbb{P}(x \in (0, x_1)) + \mathbb{P}(x\in (x_2,\pi )) \\ & = \int_0^{x_1} f_X(x) dx + \int_{x_2}^{\pi } f_X(x) dx \\ & = \left( \frac{x_1}{\pi } \right)^2 + 1 - \left( \frac{x_2}{\pi } \right)^2 \end{align*} $$ So we have $$ h_Y(y) = \frac{d}{dy} F_Y(y) = \frac{d}{dy} \mathbb{P}(Y \leq y) = \frac{d}{dy} \left[ \left( \frac{\sin^{-1}(x_1)}{\pi } \right)^2 + 1 - \left( \frac{\sin^{-1}(x_2)}{\pi } \right)^2 \right] $$ and which simplifies to $$ h_Y(y) = \begin{cases} \displaystyle{\frac{2}{\pi \sqrt{1-y^2}}} : 0<y<1 \\ \\0 : \text{otherwise} \end{cases}. $$
Recommended citation: Jiajun Zhang, (2024) Random Variables and Distributions