MLE and method of moments

In one sentence

The MLE maximises the likelihood, is invariant under reparametrisation, is asymptotically efficient — and is very often biased, sometimes on the boundary, and occasionally not unique.

Why the exam asks it

Because "maximum likelihood" sounds like it should be optimal in every sense, and it is optimal in exactly one asymptotic sense. The finite-sample failures are small and concrete: $U(0,\theta)$ and the normal variance between them cover most of what is asked.

The idea

Maximum likelihood

Maximise $L(\theta)=\prod f(x_i;\theta)$, usually by setting $\frac{d}{d\theta}\log L=0$ and checking it is a maximum.

Invariance: the MLE of $g(\theta)$ is $g(\hat\theta)$, for any $g$. This is the property MLE has and unbiasedness does not — $E[g(T)] \ne g(E[T])$ in general, so an unbiased estimator of $\theta$ gives a biased one of $\theta^2$.

Asymptotics: under regularity, the MLE is consistent, asymptotically normal with variance $1/(nI(\theta))$, and therefore asymptotically efficient — it attains the Cramér–Rao bound in the limit even when it misses it at finite $n$.

Where it misbehaves

Biased. For $N(\mu,\sigma^2)$ the MLE of the variance is $\hat\sigma^2 = \frac1n\sum(X_i-\bar X)^2$, with $E[\hat\sigma^2] = \frac{n-1}{n}\sigma^2$. Too small, systematically. The unbiased version divides by $n-1$, and that is where the familiar denominator comes from — one degree of freedom spent estimating the mean.

On the boundary. For $U(0,\theta)$ the likelihood is $\theta^{-n}$ on $\theta\ge X_{(n)}$ and zero below, so it is maximised at the smallest allowed $\theta$: $\hat\theta = X_{(n)}$. No derivative is involved — differentiating gives nothing, and a question that expects a stationary point is testing whether you noticed. It is biased low, with $E[X_{(n)}] = \frac{n}{n+1}\theta$.

Not unique. For $U(\theta,\theta+1)$ any $\theta$ in $[X_{(n)}-1, X_{(1)}]$ maximises the likelihood — an interval of MLEs.

Method of moments

Equate sample moments to population moments and solve. For one parameter, set $\bar X = E_\theta[X]$.

Simple, requires no maximisation, and consistent under mild conditions. Usually less efficient than the MLE, and can produce estimates outside the parameter space — a method-of-moments variance estimate can come out negative.

The two agree in some models: for the exponential both give $\hat\lambda = 1/\bar X$. For $U(0,\theta)$ they differ sharply — the moment estimator is $2\bar X$, which can be less than the largest observation, giving an estimated range that excludes data you have already seen. That is the cleanest argument for preferring the MLE here.

Comparing estimators

$$\text{MSE} = \text{bias}^2 + \text{variance}.$$

A biased estimator can beat an unbiased one on MSE, which is why unbiasedness is a convention rather than a requirement. For $U(0,\theta)$ the MLE $X_{(n)}$ is biased and the rescaled $\frac{n+1}{n}X_{(n)}$ is the UMVUE — and $\frac{n+2}{n+1}X_{(n)}$ beats both on MSE.

Where intuition breaks

"The MLE is unbiased." Frequently not. The normal variance MLE and $U(0,\theta)$ are the two standard cases.

"The MLE is found by differentiating." Only when the maximum is interior. For $U(0,\theta)$ it is at the boundary and the derivative never vanishes.

"The MLE is unique." $U(\theta,\theta+1)$ has an interval of them.

"MLE and method of moments agree." Sometimes, as for the exponential; not in general. For $U(0,\theta)$ the moment estimator can be smaller than an observed value.

"Unbiased is better than biased." MSE combines bias and variance, and biased estimators often win.

"Invariance holds for unbiased estimators too." It is specific to the MLE. Unbiasedness is destroyed by non-linear transformation.

"Asymptotic efficiency means efficient." In the limit. At finite $n$ the MLE may be beaten, and often is.

The exam's angle

1. Write the log-likelihood and check whether the maximum is interior. A support depending on $\theta$ signals a boundary solution.
2. Use invariance to get the MLE of any function without re-maximising.
3. Check bias by computing the expectation — for order statistics use $E[X_{(n)}] = n\theta/(n+1)$.
4. For method of moments, match the lowest moments needed and expect a simpler, weaker answer.
5. If asked which is better, ask by what criterion — bias, variance, MSE and asymptotic efficiency give different answers.

The night before

• MLE maximises $L$; invariant: MLE of $g(\theta)$ is $g(\hat\theta)$.
• Consistent, asymptotically normal with variance $1/(nI(\theta))$, asymptotically efficient.
• Biased in general: normal variance MLE has expectation $\frac{n-1}{n}\sigma^2$.
• Boundary: $U(0,\theta)$ gives $\hat\theta = X_{(n)}$, no derivative, biased low with $E = \frac{n}{n+1}\theta$.
• Non-unique: $U(\theta,\theta+1)$.
• Method of moments: simpler, consistent, usually less efficient, can leave the parameter space.
• $\text{MSE} = \text{bias}^2+\text{variance}$; biased estimators can win.