Moments of a Distribution Function

Definition Let $X$ be a random variable of the discrete type (continuous type) with probability mass function (probability density function) defined as $p_k := \mathbb{P}{ X = x_k }, k = 1,2,\cdots$ ,$(f_X)$, then if

We then say that the expected value of $X$ exists, and denote

Remark: Sometimes $\sum x_k p_k$ converges but not for $\sum \vert x_k \vert p_k$, in this case we say $\mathbb{E}(X)$ doesn’t exist. The same idea applies for $\displaystyle{\int_{\mathbb{R}} x f_X(x) dx}$.

Corollary

Suppose $X$ is a random variable and $\mathbb{E}(X)$ exists, then if $a,b \in \mathbb{R}$ and $\mathbb{E}(\vert aX+b\vert) <+\infty$, we then have $\mathbb{E}(aX+b) = a\mathbb{E}(X) + b$

Theorem

If $X$ is a random variable of the discrete type, $g$ is a Borel measurable function on $\mathbb{R}$, if $Y = g(X)$, then

On the other hand, if $X$ is a random variable of the continuous type with probability density function $f$, and $g$ is a Borel measurable function, then denote $Y = g(X)$ and $\mathbb{E}(\vert g(x) \vert) <+\infty$, where

Example: Suppose $X$ has the uniform distribution on the first $N$ natural numbers, i,e $\mathbb{P}(X = k) = \displaystyle{\frac{1}{N}}$ where $k =1,2,\cdots,N$, then

Example: Let $X$ be an random variable with probability density function defined by

Then

However

Theorem

Let $X$ be a random variable, if $\mathbb{E}( \vert X \vert^t)$ exists for some $t>0$, then for any $s \in (0,t)$, $\mathbb{E}(\vert X \vert^s)$ exists.

Proof

We will prove for continuous random variable. Let $f$ be the PDF of the random variable $X$, then

A discrete random variable can be proven in a similar way.

QED

Theorem

Let $X$ be an random variable, if $\mathbb{E}(\vert X \vert^k)$ exists for some $k>0$, then

Proof We will prove for the case when $X$ is of the continuous type. A similar proof can be used for discrete type. Denote $f_X(x)$ to be the probability density function of $X$, then

Then

QED

Probabilities of the type $\mathbb{P}{ \vert X \vert > n}$ or either of its components are called tail probabilities.

Remark: The converse of theorem 24 is not necessarily true.

Theorem

Let $X$ be a non-negative random variable with distribution function $F_X(x)$, then

Proof

Since $X$ is non-negative, then

According to integration by parts, we have

As $n \to \infty$, $-n[1-F_X(n)] \to 0$ and hence we finished the proof.

QED

Corollary

For any random variable $X$, $\mathbb{E}(X)<+\infty$ if and only if both $\displaystyle{\int_{-\infty}^0 \mathbb{P}{ X \leq x }dx}$ and $\displaystyle{\int_0^{\infty} \mathbb{P} { X \leq x } dx}$ converge.

Corollary

Let $X$ be a random variable, $\alpha>0$, then

Theorem

Let $X$ be a random variable satisfying $\lim_{n \to \infty} n^{\alpha} \mathbb{P} { \vert x \vert > n } = 0$ with $\alpha >0$, then for any $0<\beta<\alpha$, we have $\mathbb{E} ( \vert X \vert^{\beta}) < +\infty$.

Proof

Given any $\epsilon >0$, we can always choose a large enough $N$ such that $\forall n \geq N$,

and

also

Now we have

QED

Definition

Let $k$ be a positive integer and $c$ be a constant, if $\mathbb{E}(X -c)^k$ exists, we then call it the moment of order $k$ about the point $c$. If we take $c = \mathbb{E}(X) = \mu_X$, then we call $E(X - \mu_X)^k$ to be the central moment of order $k$ about the mean. Generally $\mathbb{E}(X^n)$ is the $n$th moment of $X$ and $\mathbb{E}\vert X \vert^{\alpha}$ is the $\alpha$th absolutely moment of $X$.

We will mainly study one special case, that is when $k=2$:

Definition

Let $X$ be a random variable, if $\mathbb{E}(X^2)$ exists, we denote $\mathbb{E}(X - \mathbb{E}(X))^2$ to be the variance of $X$, written as $\sigma_X^2 = \mathbf{Var}(X) = \mathbb{E}(x - \mu_X)^2$.

Remark: We know that

Corollary

Let $X$ be a random variable, $a,b \in \mathbb{R}$. Then

Definition

Let $X$ be a random variable and $\mathbb{E} \vert X \vert^2 < +\infty$, then define

to be the standardized random variable, and $\mathbb{E}(Z) = 0, \mathbf{Var}(Z) = 1$.

Generating Functions

Here, we will introduce and discuss moment generating functions (MGFs). Moment generating functions are useful for several reasons, one of which is their application to analysis of sums of random variables. Before discussing MGFs, let’s define moments.\

Definition

The $n$th moment of a random variable $X$ is dedfined to be $\mathbb{E}[X^n]$. The $n$th central moment of $X$ is defined to be $\mathbb{E}[ (X - \ \mathbb{E} X)^n]$.

Definition

The function $P(s) = \sum_{k=0}^{\infty} p_k s^k $ where $ { p_k }$ is the probability mass function of a discrete random variable $X$, is called the probability generating function (PGF). It’s clear that $P(s)$ exists for $\vert s \vert < 1$

Corollary

For the probability generating function $P(s)$ of the discrete type, we have

Also, in $P(s)$, we notice that

and

Example: Consider $X \sim Possion(\lambda)$, where

Then we know that

Thus

and

Thus we know that

and

Definition

Let $X$ be a random variable, the function

is known as the moment generating function (MGF) of the continuous type of random variable $X$, if $\mathbb{E}[e^{SX}]$ exists in some neighborhood of the origin. For discrete type, we have

Example: For each of the following random variables, find the MGF:

(a) $X$ is a discrete random variable with pmf $P_{X}(k) = \begin{cases} \frac{1}{3} : k=1 \ \frac{2}{3} : k=2\end{cases}$;\

(b) $Y$ is a $Uniform(0,1)$ random variable.

Solution:

(a) We have

(b) We have

Example: Suppose $X \sim \mathbb{E}xp(\lambda), \lambda > 0$ which is given by

In this case

and

We will introduce two theorems, however we will omit the proof:

Theorem

The moment generating function uniquely determines a distribution function and conversely, if the moment generating function exists, then it is unique, i.e the map

is bijective.

Theorem

If the moment generating function $M_X(s)$ of a random variable $X$ exists for some $\vert s \vert < s_0$, then the derivatives of all order exist at $s=0$, and

\(M_X^{(k)}(s)\Bigg\vert_{s=0} = \mathbb{E}(X^k)\) for positive integers $k$.

Why is the MGF useful? There are basically two reasons for this. First, the MGF of $X$ gives us all moments of $X$ . That is why it is called the moment generating function. Second, the MGF (if it exists) uniquely determines the distribution. That is, if two random variables have the same MGF, then they must have the same distribution. Thus, if you find the MGF of a random variable, you have indeed determined its distribution. We will see that this method is very useful when we work on sums of several independent random variables. Let’s discuss these in detail.

Recall that

where we have

Thus we have

So, we conclude that the $k$th moment of $X$ is the coefficient of $\frac{s^k}{k!}$ in the Taylor series of $M_X(s)$. If we have the Taylor series of $M_X(s)$, we can then obtain all moments of $X$.

Example: If $Y \sim Uniform(0,1)$, find $\mathbb{E}[Y^k]$.

Solution: Recall that

Also

Thus the coefficient of $\frac{s^k}{k!}$ in the Taylor series for $M_Y(s)$ is $\frac{1}{k+1}$, thus

Corollary

We can also obtain all moments of $X^k$ from its MGF:

Example: Let $X \sim Exponential(\lambda)$, find all of its moments $\mathbb{E}[X^k]$.

Solution: Recall that

So

And thus we conclude that

Example: Suppose $X \sim Geometric(p)$ which is given by

then

Moments Inequalities

We shall discuss several moments inequalities in this section.

Theorem (Markov’s Inequality)

Let $h(x)$ be a non-negative function of a random variable $X$, if $\mathbb{E}(X)$ exists, then for every $\epsilon >0$, we have

Proof

Define $A:={ x : h(x) \geq \epsilon}$ then define $f_X(x)$ to be the probability density function, then we have

QED

Corollary

Let $h(x) = \vert x \vert^r, \epsilon = k^r$ where $r>0,k>0$, then

Corollary

Let $h(x) = (x - \mu_X)^2, \epsilon = k^2 \sigma_X^2$ where $\mu_X = \mathbb{E}(X)$ and $\sigma_X^2 = \mathbf{Var}(X)$, then

if we choose $k=3$, then we get

it will also result in another inequality:

Theorem (Chebyshev’s Inequality)

If $X$ is any random variable, then for any $b>0$ we have

Proof

According to Markov’s Inequality, we have

QED

Here are some extensions of Chebyshev’s Inequality, namely Gauss Inequality and Vysochanskij–Petunin Inequality.

Corollary (Gauss Inequality)

Assume $X \sim f$ where $f$ is uni modal (meaning that $f$ obtain a single maximum) with modal $\nu$ where

and define $\tau^2 = \mathbb{E}(X - \nu)^2$, then we have

Corollary (Vysochanskij–Petunin Inequality)

Let $X \sim f$ where $f$ is uni modal and define $\xi^2 = \mathbb{E}[(x - \alpha)^2]$ for any $\alpha \in \mathbb{R}$, then $\forall \epsilon >0$, we have

The proof for corollary 13 and 14 will not be proved, since those require some work.

Theorem (Chernoff Bounds)

Let $X$ be a random variable, then $\forall a \in \mathbb{R}$,

and

Where $M_X(s)$ denotes the moment generating function of the random variable $X$.

Proof

For $t>0$, we have

A similar proof can be done when $t<0$.

QED

Corollary

Let $Z$ to be the standard normal random variable, then for $a>0$, we have

and similarly, for $a<0$, we have

Corollary

Let $X$ be a random variable with $\mathbb{E}(X) = 0$ and $\mathbf{Var}(X) = \sigma_X^2$, then

and

Proof

Proof is left as an exercise : )

QED

Theorem (Minkouski Inequality)

Let $X,Y$ be random variables, $r>0$. If $\mathbb{E} X^r, \mathbb{E} Y^r$ exist, then so is $\mathbb{E}\vert X + Y \vert^r$ and

where $C_r = \begin{cases} 1 : 0 \leq r \leq 1 \ 2^{r-1} : r > 1 \end{cases}$.

Recommended citation: Jiajun Zhang, (2024) Moments, Generating Functions and Inequalities