Joint distributions, transformations, order statistics

In one sentence

Independence means the joint density factors, uncorrelated is strictly weaker except for jointly normal variables, and order statistics come from one observation — the maximum is below $x$ exactly when all of them are.

Why the exam asks it

Because "uncorrelated therefore independent" is the standard error and has a one-line counterexample, and because the bivariate normal is the exception that makes it tempting.

The idea

Joint, marginal, conditional

$f_X(x)=\int f(x,y)\,dy$ is the marginal; $f_{Y\mid X}(y\mid x)=f(x,y)/f_X(x)$ the conditional.

$X$ and $Y$ are independent iff $f(x,y)=f_X(x)f_Y(y)$ — for all $x,y$. A useful shortcut: if the support is not a rectangle, they cannot be independent, whatever the formula looks like. A density $f(x,y)=2$ on $0<x<y<1$ has a triangular support, so $X$ and $Y$ are dependent immediately.

Marginals do not determine the joint. Different couplings of the same marginals have different dependence structures, which is what copulas describe.

Covariance and correlation

$$\operatorname{Cov}(X,Y)=E[XY]-E[X]E[Y], \qquad \rho=\frac{\operatorname{Cov}}{\sigma_X\sigma_Y}\in[-1,1].$$

Independent $\Rightarrow$ uncorrelated. The converse is false.

Let $X$ be uniform on $(-1,1)$ and $Y=X^2$. Then $E[X]=0$ and $E[XY]=E[X^3]=0$ by symmetry, so $\operatorname{Cov}=0$ and $\rho=0$. But $Y$ is a function of $X$ — knowing $X$ determines $Y$ exactly. Correlation sees only the linear part of a relationship, and there is none here.

$|\rho|=1$ happens exactly when $Y$ is an affine function of $X$ almost surely.

The exception: if $(X,Y)$ is jointly normal, then $\rho=0$ does imply independence. Note the hypothesis is joint normality — two marginally normal variables that are not jointly normal can be uncorrelated and dependent, and that distinction is examined.

Transformations

For $Y=g(X)$ with $g$ monotone, $f_Y(y)=f_X(g^{-1}(y))\left|\frac{d}{dy}g^{-1}(y)\right|$.

In several variables the factor is the absolute value of the Jacobian determinant. Two standard uses: the Box–Muller transform producing normals from uniforms, and the convolution formula $f_{X+Y}(z)=\int f_X(x)f_Y(z-x)\,dx$ for independent summands.

The absolute value is not decorative — a decreasing transformation gives a negative derivative and a density must be non-negative.

Order statistics

For $X_1,\dots,X_n$ iid with CDF $F$, sort them as $X_{(1)}\le\cdots\le X_{(n)}$.

• $F_{X_{(n)}}(x) = F(x)^n$ — the maximum is below $x$ exactly when all are.
• $F_{X_{(1)}}(x) = 1-(1-F(x))^n$ — the minimum exceeds $x$ when all do.
• In general $f_{X_{(k)}}(x) = \dfrac{n!}{(k-1)!(n-k)!}F(x)^{k-1}(1-F(x))^{n-k}f(x)$.

For uniforms on $(0,1)$, $X_{(k)}\sim\text{Beta}(k, n-k+1)$ with mean $\frac{k}{n+1}$. So the maximum of three has mean $3/4$ and the minimum $1/4$ — the order statistics divide the interval into $n+1$ equal expected gaps, which is worth remembering as a picture.

Where intuition breaks

"Uncorrelated implies independent." $Y=X^2$ with $X$ symmetric. Correlation is a linear measure.

"Jointly normal is the same as both being normal." Two normal marginals can have a non-normal joint distribution, and then $\rho=0$ does not give independence. Joint normality is the hypothesis.

"The marginals determine the joint distribution." They do not. Many joints share the same marginals.

"Independence can be checked from the formula alone." Check the support first. A non-rectangular support settles it without any algebra.

"The Jacobian factor is the derivative." Its absolute value, and in several variables the absolute determinant.

"The maximum of $n$ uniforms has mean $1/2$." $n/(n+1)$. It drifts to $1$, as it must.

"Order statistics are independent." They are strongly dependent — they are sorted. $X_{(1)}\le X_{(2)}$ always.

The exam's angle

1. Check the support before computing anything — a non-rectangular region rules out independence in one line.
2. For uncorrelated-but-dependent, reach for a symmetric $X$ with $Y=X^2$.
3. If the question says jointly normal, $\rho=0$ gives independence — and if it says only that each is normal, it does not.
4. For order statistics, write the CDF first. Maximum and minimum come straight from "all of them", and the general density follows from the multinomial count.
5. Remember $E[X_{(k)}]=k/(n+1)$ for uniforms, which answers many questions directly.

The night before

• Independent $\iff$ the joint factors everywhere; a non-rectangular support forbids it.
• Independent $\Rightarrow$ uncorrelated; converse false — $Y=X^2$ with $X$ symmetric.
• $\rho=0$ gives independence only for jointly normal variables, not for merely normal marginals.
• $|\rho|=1$ iff $Y$ is affine in $X$ almost surely.
• Transformations: multiply by the absolute Jacobian determinant.
• $F_{\max}=F^n$, $F_{\min}=1-(1-F)^n$; $f_{X_{(k)}}\propto F^{k-1}(1-F)^{n-k}f$.
• Uniform order statistics: $X_{(k)}\sim\text{Beta}(k,n-k+1)$, mean $k/(n+1)$.