Gauss–Markov, regression, ANOVA basics

Gauss–Markov

For $Y = X\beta + \varepsilon$ with $E[\varepsilon] = 0, Cov(\varepsilon) = \sigma^{2}I$ and X of full rank, the OLS estimator $\beta$̂ = (XᵀX)⁻$^{1}X^{T}Y$ is the BLUE. Normality is not needed for this — only for the t and F distributions.

Heteroscedasticity: if $Var(\varepsilon_{i}) = \sigma^{2}x_{i}^{2}$, divide through by $x_{i}$; the BLUE of $\beta$ in $Y_{i} = \beta{}x_{i} + \varepsilon_{i}$ becomes $(1/n)\sum(Y_{i}/x_{i})$, not the OLS formula.

Distribution theory (normal errors)

$- \beta$̂ ~ $N(\beta, \sigma^{2}(X^{T}X)$⁻$^{1})$.

• RSS$/\sigma^{2} = (Y - X\beta$̂$)^{T}(Y - X\beta$̂$)/\sigma^{2}$ ~ $**\chi^{2}_{n-p-1}**$, independent of $\beta$̂ (and of $X\beta$̂).
• Total and regression sums of squares are non-central $\chi^{2}$ unless the corresponding parameters vanish.

$R^{2}$ and adjusted $R^{2}$

$R^{2} =$ SSR/SST; adjusted $R^{2} = 1 - (1 - R^{2})(n-1)/(n-p-1)$. Adjusted $R^{2}$ can decrease when a useless predictor is added — that is its point.

ANOVA and estimability

One-way model $Y_{ij} = \mu + \alpha_{i} + \varepsilon_{ij}$ is overparametrised: only functions of the cell means $\mu + \alpha_{i}$ are estimable.

Function	Estimable?
$\mu + \alpha_{1}$	✓
$\alpha_{1} - \alpha_{2} ($any contrast)	✓
$2\mu + \alpha_{1} + \alpha_{2}$	✓
$\mu$ alone	✗
$\alpha_{1}$ alone	✗
$\mu + \alpha_{1} + \alpha_{2}$	✗

A linear function $c^{T}\beta$ is estimable ⇔ c is in the row space of X.

Design degrees of freedom

Design	Error d.f.
CRD, k treatments, N units	N − k
RBD, k treatments, r blocks	(k−1)(r−1)
LSD, p × p	(p−1)(p−2)

Key takeaways

• Gauss–Markov needs equal variances; otherwise weight by 1/variance.
• RSS$/\sigma^{2}$ is central $\chi^{2}$ with n − p − 1 d.f.; the other sums of squares are generally non-central.
• In ANOVA only contrasts and cell means are estimable.