Uniformly Minimum Variance Unbiased Estimator (UMVUE)

Now we would like to focus on the class of unbiased estimators of a real valued parameter $\tau(\theta)$, $\tau : \Theta \to \mathbb{R}$. We would like to introduce the method of Uniformly Minimum Variance Unbiased Estimator (UMVUE):

Let $\vec X = \begin{pmatrix} X_1 \ X_2 \ \vdots \ X_n \end{pmatrix}$ be a random vector with a joint pdf (pmf) given by $p_{\theta}(x_1,\cdots,x_n) = p_{\theta}(\vec x)$, and the vectoer $\vec x$ given by $\vec x = \begin{pmatrix} x_1 \ x_2 \ \vdots \ x_n \end{pmatrix}$.

How do we actually find a UMVUE? There are two ways, one is by Cramér-Rao Lower Bound, one is by Rao-Blackwell \& Lehmann-Scheffé theorem. We first introduce the Cramér-Rao Lower Bound (CRLB). Assume $d=1$, $\Theta \subset \mathbb{R}$, $\vec X \sim p_{\theta}$ (joint pdf / pmf).

• Assume that the family ${ p_{\theta} : \theta \in \Theta \subset \mathbb{R} }$ has a common support, i.e

\(\mathcal{S} := \left\{ \vec x = \begin{pmatrix} x_1 \\ \vdots \\ x_n \end{pmatrix} : p_{\theta}(\vec x) > 0 \right\}\) and does not depend on $\theta$. (In this case $Uniform(0,\theta)$ is excluded);

• Furthermore, assume for $\vec x \in \mathcal{S}$, $\theta \in \Theta \subset \mathbb{R}$, $\displaystyle{\frac{d}{d\theta}\log p_{\theta}(\vec x)}$ exists;

• Also, assume for any statistics $h(\vec X)$ with $\mathbb{E}{ \vert h(\vec X) \vert } < +\infty, \forall \theta \in \Theta$ we have

whenever the $R.H.S$ is finite.

Here, we denote

it is called the Fisher Information.

Note that if we have $X_1,\cdots,X_n \overset{i.i.d}{\sim} f(x,\theta)$, then by definition

hence in this case

and now

sometimes it is easier to compute the Fisher information for one sample, we may do so and multiply by $n$ in the end, with the assumption that we are working with random sample (i.i.d).

Example: Suppose we have a random sample $X_1,\cdots,X_n \overset{i.i.d}{\sim} Bernoulli(\theta)$, its probability mass function is given by $f(x,\theta) = \theta^x (1-\theta)^{1-x}, x \in {0,1}$ and $\theta \in (0,1)$. Find the CRLB of all unbiased estimator of $\tau(\theta) = \theta$.

Solution: We first find the Fisher information. We investigate the function

and

by Bartlett’s identity, we have

Now $\tau(\theta) = \theta$ so $[\tau’(\theta)]^2 = 1$ hence the CRLB is given by

An unbiased estimator of $\tau(\theta)$ might be the UMVUE, but its variance could be larger than the CRLB. That is, CRLB is not a sharp lower bound.

Example:

We say $f_\theta(x)$ belongs to exponential family (note that you may find me use $f_\theta(x), f(x,\theta), f(x \vert \theta)$, they are all the same notation, meaning $\theta$ is our parameter, but the density function is a function of $x$), if its density function takes the form

for some non-negative function $h(x)$ of $x$ and $c(\theta)$ of $\theta$, and $T(x)$ is a function of $x$, also its support does not depend on $\theta$. Then show that if a random sample $X_1,\cdots,X_n \overset{i.i.d}{\sim} f_\theta(x)$ where $f_\theta(x)$ belongs to exponential family, then

is the UMVUE of $\tau(\theta) = -c’(\theta) / (c(\theta) \omega’(\theta))$.

Solution: Fist of all, the joint pdf of the sample takes the form

so we have

By the previous theorem, it suffies to check $T(\vec X)$ is unbiased. Using Bartlett’s identity, we know that

which means

hence $T(\vec X) = \displaystyle{\frac{1}{n} \sum_{i=1}^n T(X_i)}$ is unbiased, and by the previous theorem, it is the UMVUE.

Example: Let a random sample $X_1,\cdots,X_n \overset{i.i.d}{\sim} Poisson(\theta)$ with density $\displaystyle{f(x,\theta) = e^{-\theta} \frac{\theta^x}{x!}}$, $x \in \mathbb{N}_0$.

• find the UMVUE of $\theta$.

• Find the CRLB of all estimates of $\theta$.

Solution:

We rewrite the density of Poisson distribution as

let $h(x) = 1/x! , c(\theta) = e^{-\theta}, \omega(\theta) = \log(\theta), T(x) = x$, and its support $x \in \mathbb{N}_0$ does not depend on $\theta$, so it belongs to exponential family, then we use the result from the previous example, now $\tau(\theta) = \theta$, so the UMVUE is just the sample mean, given by

By direct computation, we have

so using Bartlett’s identity, the Fisher information of one sample is given by

since $X \sim Poisson(\theta)$, hence the CRLB can be computed by

Recommended citation: Jiajun Zhang, (2025) Uniformly Minimum Variance Unbiased Estimator (UMVUE)