Neyman–Pearson lemma and UMP tests

In one sentence

Fix the type I error rate, then maximise power — and the Neyman–Pearson lemma says the likelihood ratio does it for a simple alternative, while for two-sided alternatives no uniformly best test exists at all.

Why the exam asks it

Because the asymmetry between the two errors is the design of the whole framework and is routinely misread, and because "UMP test" has precise existence conditions that fail in the most familiar case.

The idea

The two errors

• Type I: reject a true $H_0$. Probability $\alpha$, the size or level.
• Type II: fail to reject a false $H_0$. Probability $\beta$.
• Power $=1-\beta$, a function of the true parameter.

The framework fixes $\alpha$ and maximises power. That asymmetry is deliberate: $H_0$ is the default, and rejecting it requires evidence. It is also why "do not reject" is not "accept" — the type II rate is uncontrolled, so failing to reject may mean the null is true or merely that the test lacked power.

Neyman–Pearson

For simple against simple, $H_0:\theta=\theta_0$ against $H_1:\theta=\theta_1$:

The most powerful test of size $\alpha$ rejects when

$$\frac{L(\theta_1)}{L(\theta_0)} > k,$$

with $k$ chosen to give size $\alpha$.

That is the whole lemma, and everything else in the topic is an attempt to extend it.

For $N(\mu,1)$ testing $\mu=0$ against $\mu=1$, the ratio is increasing in $\bar X$, so the test rejects for large $\bar X$ — and the region does not depend on which $\mu_1>0$ was chosen. That observation is what produces UMP tests.

UMP tests and their limits

A test is uniformly most powerful if it is most powerful against every alternative in $H_1$ simultaneously.

Karlin–Rubin: if the family has monotone likelihood ratio in $T$, then rejecting for large $T$ is UMP for $H_0:\theta\le\theta_0$ against $H_1:\theta>\theta_0$. One-sided alternatives in the exponential family are exactly the well-behaved case.

For two-sided alternatives a UMP test generally does not exist. The most powerful test against $\mu>0$ rejects in the upper tail; against $\mu<0$ it rejects in the lower. No single region is best against both, so no UMP test exists.

The repair is to narrow the class: among unbiased tests — those whose power never falls below $\alpha$ — the two-sided symmetric test is UMP unbiased. "UMPU" exists where "UMP" does not, and that distinction is examined.

Discrete statistics and randomisation

With a discrete statistic the attainable sizes are a finite set. For $\text{Bin}(10,\tfrac12)$ the upper tail masses jump from $1/1024$ to $11/1024$, so an exact size of $0.05$ is unavailable.

Randomised tests reject with a probability strictly between $0$ and $1$ on the boundary value, achieving exactly $\alpha$. They are theoretically necessary for the lemma to be exactly true and are not used in practice, which is why they appear in examinations and nowhere else.

Where intuition breaks

"Failing to reject means $H_0$ is true." It means the data did not rule it out. The type II rate is not controlled by the design.

"A UMP test always exists." Not for two-sided alternatives. Restricting to unbiased tests recovers one.

"The most powerful test is the one with smallest $\alpha$." $\alpha$ is fixed by choice; power is what is maximised. Lowering $\alpha$ lowers power.

"$p$-value is the probability the null is true." It is $P(\text{data this extreme}\mid H_0)$ — a probability about data, not about hypotheses. Reversing the conditional is the same error as the medical test in the Bayes chapter.

"Power depends only on the test." On the test, the sample size, the level, and the true parameter value. Quoting power without an alternative is meaningless.

"Neyman–Pearson applies to composite hypotheses." It is stated for simple against simple. Karlin–Rubin extends it one-sidedly under MLR.

"Randomised tests are a technicality with no consequence." They are what make exact sizes attainable for discrete data, and questions do exploit that.

The exam's angle

1. Form the likelihood ratio and simplify until the rejection region is a statement about a statistic — usually $\bar X$ or $\sum X_i$.
2. Check whether the region depends on $\theta_1$. If not, the test is UMP one-sided.
3. For two-sided, expect no UMP and say so, or move to the unbiased class.
4. Compute power as a probability under the alternative, not under the null.
5. For discrete statistics, list the attainable sizes before claiming a test has size $\alpha$.

The night before

• Type I $=\alpha$, fixed. Type II $=\beta$, power $=1-\beta$, uncontrolled. "Do not reject" is not "accept".
• Neyman–Pearson: for simple against simple, reject when $L(\theta_1)/L(\theta_0)>k$.
• Karlin–Rubin: MLR gives a UMP test for one-sided alternatives.
• No UMP for two-sided in general; UMP unbiased exists.
• $p$-value is $P(\text{data}\mid H_0)$, not $P(H_0\mid\text{data})$.
• Discrete statistics need randomised tests for exact size.
• Power depends on the alternative, the level and $n$.