The solution: hold out data

• Cross-validation repeatedly splits the data into a training set and a validation set.
• Model error is evaluated on observations the model has not seen during fitting.
• This gives an honest estimate of out-of-sample (generalisation) error.

Cross-validation is also our main tool for choosing the regularization hyperparameter \lambda — we will return to it in the regularization section.

• k = 5 or k = 10 are standard choices — they balance bias and variance of the CV estimate well.
• k = n (LOOCV): leave one observation out at a time — low bias, but high variance and expensive.

For OLS, there is a remarkable shortcut that avoids refitting the model n times:

\text{LOOCV} = \frac{1}{n}\sum_{i=1}^n \left(\frac{e_i}{1 - h_{ii}}\right)^2,

where e_i = y_i - \hat{y}_i is the ordinary residual and h_{ii} is the i-th diagonal of the hat matrix \mathbf{H}.

Why does this work?

• When observation i is removed, the prediction at x_i changes by e_i / (1 - h_{ii}).
• Observations with high leverage h_{ii} have a large influence on the LOOCV score — they are “risky” in the sense that leaving them out changes the fit substantially.
• This formula is computationally free once \mathbf{H} is computed.

The regularization idea

• Add a penalty on the size of \boldsymbol{\beta} to the least squares objective.
• Trading a little bias for a large reduction in variance can lower total test error.

This is exactly the bias-variance tradeoff in action: we deliberately introduce bias to tame variance.

Regularization can be written in two equivalent forms.

Penalty (Lagrangian) form — what we minimize:

\min_{\boldsymbol{\beta}}\; \|\mathbf{y} - \mathbf{X}\boldsymbol{\beta}\|^2_2 + \lambda\|\boldsymbol{\beta}\|_q

Constraint form — equivalent formulation:

\min_{\boldsymbol{\beta}}\; \|\mathbf{y} - \mathbf{X}\boldsymbol{\beta}\|^2_2 \quad \text{subject to} \quad \|\boldsymbol{\beta}\|_q \leq t

• For every value of \lambda \geq 0 there is a corresponding budget t \geq 0 that produces the same solution.
• As \lambda increases, t decreases — the constraint tightens and coefficients must shrink further.
• At \lambda = 0 (t = \infty): no constraint — recovers the OLS solution.
• At \lambda \to \infty (t = 0): all \hat{\beta}_j = 0 — the null model.

The constraint perspective makes the geometry transparent: the regularized solution is the point on the constraint region \|\boldsymbol{\beta}\|_q \leq t that is closest to the OLS solution.

The regularization penalty \lambda \sum_j \beta_j^2 (or \lambda \sum_j |\beta_j|) penalizes coefficients by their magnitude.

The scale problem

• Suppose X_1 is measured in millions of dollars and X_2 in metres.
• A unit change in X_1 moves the response by \beta_1 — but \beta_1 will naturally be tiny (because the units are huge).
• A unit change in X_2 moves the response by \beta_2 — with larger units, \beta_2 can be larger.
• The penalty then falls almost entirely on \beta_2, unfairly shrinking it relative to \beta_1.

The fix: standardize before fitting

X_j \leftarrow \frac{X_j - \bar{X}_j}{s_j}

This puts all predictors on a unit-variance, zero-mean scale so the penalty is applied equally.

Important: the intercept \beta_0 is never penalized — shrinking it would shift all predictions by the mean of y, which is not what we want.

Closed-form solution:

\hat{\boldsymbol{\beta}}_{\text{ridge}} = (\mathbf{X}^\top\mathbf{X} + \lambda \mathbf{I})^{-1}\mathbf{X}^\top\mathbf{y}.

Why does adding \lambda\mathbf{I} help? Think in terms of eigenvalues.

• \mathbf{X}^\top\mathbf{X} is a symmetric positive semi-definite matrix with eigenvalues \lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_p \geq 0.
• The OLS inverse involves 1/\lambda_j for each eigenvalue. If any \lambda_j \approx 0 (nearly singular — which happens when predictors are correlated), that term explodes, inflating \text{Var}(\hat{\boldsymbol{\beta}}).
• Adding \lambda\mathbf{I} shifts every eigenvalue up by \lambda: the eigenvalues of \mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I} are \lambda_j + \lambda.
• The inverse now involves 1/(\lambda_j + \lambda) — even if \lambda_j \approx 0, the denominator is at least \lambda > 0, so it stays bounded.

The matrix is therefore always invertible for any \lambda > 0, regardless of multicollinearity.

• Ridge shrinks all coefficients toward zero but never sets them exactly to zero — all predictors are retained.
• Particularly useful when predictors are correlated (multicollinearity).
• Predictors must be standardised first (the penalty is scale-sensitive).

• No closed-form solution — requires coordinate descent or convex optimization.
• The L_1 penalty induces sparsity: for large enough \lambda, some \hat{\beta}_j are set exactly to zero.
• Lasso performs automatic variable selection — it simultaneously shrinks and selects.
• Like Ridge, predictors must be standardised.

The key difference: Ridge shrinks coefficients toward zero uniformly; Lasso can zero them out completely.

This geometry is why L_2 (Ridge) never produces exactly-zero coefficients: the smooth L_2 ball has no corners, so the ellipse touches it at a non-axis point.

• Combines the sparsity of Lasso with the grouping property of Ridge.
• When predictors are correlated, Lasso tends to arbitrarily select one from a group; Elastic Net tends to select the group together.
• Two hyperparameters: \lambda (overall penalty strength) and \alpha (L1 vs. L2 mix).

Rules of thumb

• Use Ridge when you expect a dense signal — most predictors contribute something.
• Use Lasso when you expect sparsity — only a few predictors truly matter.
• Use Elastic Net when predictors are correlated and sparse — combines the best of both.
• When p \gg n, prefer Lasso or Elastic Net to obtain an interpretable sparse model.

We use sklearn’s built-in diabetes dataset (GeeksForGeeks 2024), a classic regression benchmark.

• n = 442 patients; response = disease progression one year after baseline (continuous).
• p = 10 predictors: age, sex, BMI, blood pressure, and six blood serum measurements.
• Predictors are already mean-centred and scaled by sklearn.

Before fitting any model we hold out 25% of the data as a test set — this will never be seen during training or hyperparameter tuning.

Because the predictors are already scaled in this dataset, we do not need to apply StandardScaler here. In general, you should always scale when using regularized methods.

We fit Ridge with alpha=1 (\lambda = 1) as a starting point, then inspect the coefficients.

We fit Lasso with alpha=1. Notice that some coefficients are set exactly to zero — Lasso has performed automatic variable selection.

Elastic Net adds a second hyperparameter l1_ratio (\alpha in our notation) controlling the L1/L2 mix. Here we use l1_ratio=0.5 — equal weight to both penalties.

alpha=1 was arbitrary. We use built-in CV classes to search over a grid and find the optimal \lambda for each method.