Linear Models and Multivariate

Gauss–Markov

For $Y=X\beta +\epsilon$ with $E[\epsilon ]=0,Cov(\epsilon )={\sigma }^{2}I$ and X of full rank, the OLS estimator $\beta$̂ = (XᵀX)⁻${}^1{X}^{T}Y$ is the BLUE. Normality is not needed for this — only for the t and F distributions.

Heteroscedasticity: if $Var({\epsilon }_i)={\sigma }^2{x}_i^2$, divide through by $x_i$; the BLUE of $\beta$ in $Y_i=\beta x_i+{\epsilon }_i$ becomes $(1/n)\sum (Y_i/x_i)$, not the OLS formula.

Distribution theory (normal errors)

$-\beta$̂ ~ $N(\beta ,{\sigma }^2(X^{T}X)$⁻${}^1)$.

• RSS$/{\sigma }^2=(Y-X\beta$̂${)}^T(Y-X\beta$̂$)/{\sigma }^2$ ~ $\ast \ast {\chi }_{n-p-1}^2\ast \ast$, independent of $\beta$̂ (and of $X\beta$̂).
• Total and regression sums of squares are non-central ${\chi }^2$ unless the corresponding parameters vanish.

$R^2$ and adjusted $R^2$

$R^2=$ SSR/SST; adjusted $R^2=1-(1-R^2)(n-1)/(n-p-1)$. Adjusted $R^2$ can decrease when a useless predictor is added — that is its point.

ANOVA and estimability

One-way model $Y_{ij}=\mu +{\alpha }_i+{\epsilon }_{ij}$ is overparametrised: only functions of the cell means $\mu +{\alpha }_i$ are estimable.

A linear function $c^T\beta$ is estimable ⇔ c is in the row space of X.

Design degrees of freedom

Key takeaways

• Gauss–Markov needs equal variances; otherwise weight by 1/variance.
• RSS$/{\sigma }^2$ is central ${\chi }^2$ with n − p − 1 d.f.; the other sums of squares are generally non-central.
• In ANOVA only contrasts and cell means are estimable.