Search for probability and statistics terms on Statlect
StatLect
Index > Fundamentals of statistics > Point estimation

Estimation of the variance

by , PhD

Variance estimation is a statistical inference problem in which a sample is used to produce a point estimate of the variance of an unknown distribution.

The problem is typically solved by using the sample variance as an estimator of the population variance.

In this lecture, we present two examples, concerning:

  1. IID samples from a normal distribution whose mean is known;

  2. IID samples from a normal distribution whose mean is unknown.

For each of these two cases, we derive the expected value, the distribution and the asymptotic properties of the variance estimator.

Table of Contents

Normal IID samples - Known mean

In this example of variance estimation we make assumptions that are similar to those we made in the mean estimation of normal IID samples.

The sample

The sample is made of n independent draws from a normal distribution.

Specifically, we observe the realizations of n independent random variables X_1, ..., X_n, all having

The estimator

We use the following estimator of variance:[eq1]

Expected value of the estimator

The expected value of the estimator [eq2] is equal to the true variance sigma^2:[eq3]

Proof

This can be proved using linearity of the expected value:[eq4]

Therefore, the estimator [eq2] is unbiased.

Variance of the estimator

The variance of the estimator [eq2] is[eq7]

Proof

This can be proved using the fact that for a normal distribution [eq8] and the formula for the variance of an independent sum:[eq9]

Therefore, the variance of the estimator tends to zero as the sample size n tends to infinity.

Distribution of the estimator

The estimator [eq2] has a Gamma distribution with parameters n and sigma^2.

Proof

The estimator [eq2] can be written as[eq12]where the variables $Z_{i}$ are independent standard normal random variables and $W$, being a sum of squares of n independent standard normal random variables, has a Chi-square distribution with n degrees of freedom (see the lecture entitled Chi-square distribution for more details). Multiplying a Chi-square random variable with n degrees of freedom by $sigma ^{2}/n$ one obtains a Gamma random variable with parameters n and sigma^2 (see the lecture entitled Gamma distribution for more details).

Risk of the estimator

The mean squared error of the estimator is[eq13]

Consistency of the estimator

The estimator[eq14]can be viewed as the sample mean of a sequence [eq15] where the generic term of the sequence is[eq16]

Since the sequence [eq15] is an IID sequence with finite mean, it satisfies the conditions of Kolmogorov's Strong Law of Large Numbers.

Therefore, the sample mean of $Y_{n}$ converges almost surely to the true mean [eq18]:[eq19]

In other words, the estimator [eq20] is strongly consistent.

It is also weakly consistent, because almost sure convergence implies convergence in probability:[eq21]

Normal IID samples - Unknown mean

This example of variance estimation is similar to the previous one. The only difference is that we relax the assumption that the mean of the distribution is known.

The sample

The sample is made of independent draws from a normal distribution.

Specifically, we observe the realizations of n independent random variables X_1, ..., X_n, all having a normal distribution with:

The estimator

In this example also the mean of the distribution, being unknown, needs to be estimated.

It is estimated with the sample mean Xbar_n:[eq22]

We use the following estimators of variance:

  1. the unadjusted sample variance:[eq23]

  2. the adjusted sample variance:[eq24]

Expected value of the estimator

The expected value of the unadjusted sample variance is[eq25]

Proof

This can be proved as follows:[eq26]But [eq27] when $i eq j$ (because X_i and $X_{j}$ are independent when $i eq j$ - see Mutual independence via expectations). Therefore,[eq28]

Therefore, the unadjusted sample variance $S_{n}^{2}$ is a biased estimator of the true variance sigma^2.

The adjusted sample variance $s_{n}^{2}$, on the contrary, is an unbiased estimator of variance:[eq29]

Proof

This can be proved as follows:[eq30]

Thus, when also the mean mu is being estimated, we need to divide by $n-1 $ rather than by n to obtain an unbiased estimator.

Intuitively, by considering squared deviations from the sample mean rather than squared deviations from the true mean, we are underestimating the true variability of the data.

In fact, the sum of squared deviations from the true mean is always larger than the sum of squared deviations from the sample mean.

Dividing by $n-1$ rather than by n exactly corrects this bias. The number $n-1$ by which we divide is called the number of degrees of freedom and it is equal to the number of sample points (n) minus the number of other parameters to be estimated (in our case 1, the true mean mu).

The factor by which we need to multiply the biased estimator $S_{n}^{2}$ to obtain the unbiased estimator $s_{n}^{2}$ is[eq31]

This factor is known as degrees of freedom adjustment, which explains why $S_{n}^{2}$ is called unadjusted sample variance and $s_{n}^{2}$ is called adjusted sample variance.

Variance of the estimator

The variance of the unadjusted sample variance is[eq32]

Proof

This is proved in the following subsection (distribution of the estimator).

The variance of the adjusted sample variance is[eq33]

Proof

This is also proved in the following subsection (distribution of the estimator).

Therefore, both the variance of $S_{n}^{2}$ and the variance of $s_{n}^{2}$ converge to zero as the sample size n tends to infinity.

Note that the unadjusted sample variance $S_{n}^{2}$, despite being biased, has a smaller variance than the adjusted sample variance $s_{n}^{2}$, which is instead unbiased.

Distribution of the estimator

The unadjusted sample variance $S_{n}^{2}$ has a Gamma distribution with parameters $n-1$ and [eq34].

Proof

To prove this result, we need to use some facts on quadratic forms involving normal random variables, which have been introduced in the lecture entitled Normal distribution - Quadratic forms. To understand this proof, you need to first read that lecture, in particular the section entitled Sample variance as a quadratic form. Define the matrix[eq35]where I is an $n imes n$ identity matrix and $imath $ is a $n imes 1$ vector of ones. $M$ is symmetric and idempotent. Denote by X the $n imes 1 $ random vector whose i-th entry is equal to X_i. The random vector X has a multivariate normal distribution with mean $mu imath $ and covariance matrix $sigma ^{2}I$.

Using the fact that the matrix $M$ is symmetric and idempotent, the unadjusted sample variance can be written as[eq36]

By using the fact that the random vector[eq37]has a standard multivariate normal distribution and the fact that $Mimath =0$, we can rewrite [eq38]In other words, $S_{n}^{2}$ is proportional to a quadratic form in a standard normal random vector ($Z^{ op }MZ$) and the quadratic form involves a symmetric and idempotent matrix whose trace is equal to $n-1$. Therefore, the quadratic form $Z^{ op }MZ$ has a Chi-square distribution with $n-1$ degrees of freedom. Finally, we can write[eq39]that is, $S_{n}^{2}$ is a Chi-square random variable divided by its number of degrees of freedom and multiplied by [eq40]. Thus, $S_{n}^{2}$ is a Gamma random variable with parameters $n-1$ and [eq41] (see the lecture entitled Gamma distribution for an explanation). Also, by the properties of Gamma random variables, its expected value is[eq42]and its variance is[eq43]

The adjusted sample variance $s_{n}^{2}$ has a Gamma distribution with parameters $n-1$ and sigma^2.

Proof

The proof of this result is similar to the proof for unadjusted sample variance found above. It can also be found in the lecture entitled Normal distribution - Quadratic forms. Here, we just notice that $s_{n}^{2}$, being a Gamma random variable with parameters $n-1$ and sigma^2, has expected value[eq44]and variance[eq33]

Risk of the estimator

The mean squared error of the unadjusted sample variance is[eq46]

Proof

It can be proved as follows:[eq47]

The mean squared error of the adjusted sample variance is[eq48]

Proof

It can be proved as follows:[eq49]

Therefore the mean squared error of the unadjusted sample variance is always smaller than the mean squared error of the adjusted sample variance:[eq50]

Consistency of the estimator

Both the unadjusted and the adjusted sample variances are consistent estimators of the unknown variance sigma^2.

Proof

The unadjusted sample variance[eq23]can be written as[eq52]where we have defined[eq53]The two sequences Xbar_n and $W_{n}$ are the sample means of X_n and $X_{n}^{2}$ respectively. The latter both satisfy the conditions of Kolmogorov's Strong Law of Large Numbers (they form IID sequences with finite means), which implies that their sample means Xbar_n and $W_{n}$ converge almost surely to their true means:[eq54]Since the function[eq55]is continuous and almost sure convergence is preserved by continuous transformations, we obtain[eq56]Therefore the estimator $S_{n}^{2}$ is strongly consistent. It is also weakly consistent because almost sure convergence implies convergence in probability:[eq57]The adjusted sample variance $s_{n}^{2}$ can be written as[eq58]The ratio $frac{n}{n-1}$ can be thought of as a constant random variable $Z_{n}$ defined as follows:[eq59]which converges almost surely to 1. Therefore, [eq60]where both $Z_{n}$ and $S_{n}^{2}$ are almost surely convergent. Since the product is a continuous function and almost sure convergence is preserved by continuous transformation, we have[eq61]Thus, also $s_{n}^{2}$ is strongly consistent.

Solved exercises

Below you can find some exercises with explained solutions.

Exercise 1

You observe three independent draws from a normal distribution having unknown mean mu and unknown variance sigma^2. Their values are 50, 100 and 150.

Use these values to produce an unbiased estimate of the variance of the distribution.

Solution

The sample mean is [eq62]An unbiased estimate of the variance is provided by the adjusted sample variance:[eq63]

Exercise 2

A machine (a laser rangefinder) is used to measure the distance between the machine itself and a given object.

When measuring the distance to an object located 10 meters apart, measurement errors committed by the machine are normally and independently distributed and are on average equal to zero.

The variance of the measurement errors is less than 1 squared centimeter, but its exact value is unknown and needs to be estimated.

To estimate it, we repeatedly take the same measurement and we compute the sample variance of the measurement errors (which we are also able to compute because we know the true distance).

How many measurements do we need to take to obtain an estimator of variance having a standard deviation less than 0.1 squared centimeters?

Solution

Denote the measurement errors by X_1, ..., X_n. The following estimator of variance is used: [eq64]The variance of this estimator is[eq65]Thus[eq66]We need to ensure that[eq67]or[eq68]which is certainly verified if[eq69]or[eq70]

How to cite

Please cite as:

Taboga, Marco (2021). "Estimation of the variance", Lectures on probability theory and mathematical statistics. Kindle Direct Publishing. Online appendix. https://www.statlect.com/fundamentals-of-statistics/variance-estimation.

The books

Most of the learning materials found on this website are now available in a traditional textbook format.

Featured pages
Explore
Main sections
About
Glossary entries
Share
  • To enhance your privacy,
  • we removed the social buttons,
  • but don't forget to share.