Gauss–Markov, regression, ANOVA basics

In one sentence

Gauss–Markov says least squares is best among linear unbiased estimators and needs no normality at all — normality is what buys the $t$ and $F$ tests, and the two claims are constantly merged.

Why the exam asks it

Because "BLUE" contains its own qualifier and people read past it, and because the assumption list has four items that do four different jobs. Options attach the wrong conclusion to the wrong assumption.

The idea

The linear model is $\mathbf{y} = X\boldsymbol\beta + \boldsymbol\varepsilon$ with $X$ an $n\times(p+1)$ design matrix of full rank.

Least squares minimises $\|\mathbf{y}-X\boldsymbol\beta\|^2$, giving

$$\hat{\boldsymbol\beta} = (X'X)^{-1}X'\mathbf{y}, \qquad \hat{\mathbf{y}} = H\mathbf{y}, \quad H = X(X'X)^{-1}X'.$$

$H$ is the hat matrix: symmetric, idempotent ($H^2=H$), and projecting onto the column space of $X$. Its trace is $p+1$, the number of parameters — which is where the degrees of freedom come from.

What each assumption buys

• $E[\boldsymbol\varepsilon]=0$ gives unbiasedness: $E[\hat\beta]=\beta$.
• $\operatorname{Var}(\boldsymbol\varepsilon)=\sigma^2I$ — homoscedastic and uncorrelated — gives the Gauss–Markov conclusion.
• $X$ full rank makes $\beta$ identifiable at all; without it $(X'X)^{-1}$ does not exist.
• Normality gives the exact $t$ and $F$ distributions.

Gauss–Markov: under the first three, $\hat\beta$ is the best linear unbiased estimator — minimum variance among estimators that are linear in $\mathbf{y}$ and unbiased.

No normality is required. Add it and more follows: $\hat\beta$ becomes the MLE and the UMVUE among all unbiased estimators, linear or not. Those are two different theorems with two different hypotheses, and conflating them is the topic's central error.

Note also what BLUE excludes. Ridge regression is biased and can have smaller mean squared error; Gauss–Markov does not contradict that, because ridge is not unbiased.

ANOVA

$$\underbrace{\sum(y_i-\bar y)^2}_{SST} = \underbrace{\sum(\hat y_i-\bar y)^2}_{SSR} + \underbrace{\sum(y_i-\hat y_i)^2}_{SSE},$$

an orthogonal decomposition — the residual vector is perpendicular to the fitted one, which is Pythagoras in $\mathbb{R}^n$ and the reason the cross term vanishes.

Degrees of freedom: $n-1 = p + (n-p-1)$. Then

$$R^2 = \frac{SSR}{SST}, \qquad F = \frac{SSR/p}{SSE/(n-p-1)} \sim F_{p,\,n-p-1}$$

under $H_0:\beta_1=\cdots=\beta_p=0$ with normal errors.

$s^2 = SSE/(n-p-1)$ is unbiased for $\sigma^2$ — the denominator is the error degrees of freedom, not $n$.

What can go wrong

Multicollinearity: near-dependent columns make $X'X$ nearly singular, so variances inflate. Estimates stay unbiased and become useless — large standard errors, unstable signs. It is a precision problem, not a bias problem.

$R^2$ never decreases when a predictor is added, however irrelevant, which is why adjusted $R^2$ penalises $p$. A high $R^2$ is not evidence of a good model.

Heteroscedasticity breaks Gauss–Markov but not unbiasedness; the fix is weighted least squares, which is BLUE under the corrected variance structure.

Where intuition breaks

"Gauss–Markov requires normal errors." It does not. Normality is needed for the $t$ and $F$ tests, not for BLUE.

"BLUE means best among all unbiased estimators." Among linear ones. With normality it upgrades to all, by a different theorem.

"An unbiased estimator cannot be beaten." On MSE it can, by biased estimators such as ridge.

"High $R^2$ means the model is good." $R^2$ rises with every added predictor. It measures fit to the data in hand, not validity.

"Multicollinearity biases the estimates." It inflates their variance. They remain unbiased.

"$s^2 = SSE/n$." $SSE/(n-p-1)$. Dividing by $n$ gives the MLE, which is biased low — the same $n$ against $n-1$ issue as for the sample variance, with more parameters.

"The $F$ test tells you which predictor matters." It tests them jointly. Individual $t$ tests address single coefficients, and $t^2=F_{1,\cdot}$ ties them together.

The exam's angle

1. Count degrees of freedom first: $p$ for regression, $n-p-1$ for error. Most ANOVA questions are that count.
2. Read which assumption an option invokes and check it against what that assumption actually buys.
3. Use idempotence of $H$ — $H^2=H$, $\operatorname{tr}H = p+1$ — for questions about projections and residuals.
4. Remember $SST=SSR+SSE$ and $R^2=SSR/SST$, and that $R^2$ is non-decreasing in $p$.
5. Separate bias from variance when diagnosing multicollinearity or shrinkage.

The night before

• $\hat\beta=(X'X)^{-1}X'y$; $H=X(X'X)^{-1}X'$ is symmetric idempotent with trace $p+1$.
• Gauss–Markov: $E[\varepsilon]=0$, $\operatorname{Var}=\sigma^2I$, $X$ full rank $\Rightarrow$ OLS is BLUE. No normality.
• With normality: OLS $=$ MLE $=$ UMVUE among all unbiased estimators.
• $SST=SSR+SSE$; df $n-1=p+(n-p-1)$; $R^2=SSR/SST$; $F=\frac{SSR/p}{SSE/(n-p-1)}$.
• $s^2=SSE/(n-p-1)$ is unbiased.
• Multicollinearity inflates variance, not bias. $R^2$ never falls when a predictor is added.