• In Lecture 7 we introduced the conditional expectation \mathbb{E}[Y \mid X = x]: the expected value of Y when X is fixed at x.
• This is the regression function — the theoretically optimal prediction of Y given X.

Why impose linearity?

• The true function \mu(x) could take any shape, but we cannot estimate it without assumptions.
• Linear regression restricts the model to \mu(x) = \beta_0 + \beta_1 x — a straight line.
• This is a modelling choice: it is interpretable, tractable with small n, and often a reasonable local approximation.

Simple vs. multiple

• Simple linear regression (SLR): one predictor X. This lecture.
• Multiple linear regression (MLR): p predictors X_1, \ldots, X_p. Lecture 11.

Linear vs. nonlinear

• “Linear” refers to linearity in the parameters \beta, not necessarily in X itself.
  • Y = \beta_0 + \beta_1 X^2 is still a linear model (we treat X' = X^2 as the predictor).
  • Y = \beta_0 \exp(\beta_1 X) is nonlinear in \beta and requires different methods.
• Much of machine learning can be viewed as flexible, nonlinear regression.

Regression in the supervised learning framework

Two components of the model

• Systematic part \beta_0 + \beta_1 x_i: the signal we want to estimate.
• Random part \varepsilon_i: the noise we want to characterise and account for.

Together, 1–4 imply that Y_i \stackrel{\text{iid}}\sim \mathcal{N} (\beta_0 + \beta_1 x_i,\, \sigma^2).

• Squaring makes the criterion differentiable and penalises large deviations heavily.
• It yields closed-form estimators — we can solve analytically by setting derivatives to zero.

Taking \partial \text{RSS}/\partial b_0 = 0 and \partial \text{RSS}/\partial b_1 = 0:

-2\sum_{i=1}^n (y_i - b_0 - b_1 x_i) = 0,\quad \& \quad -2\sum_{i=1}^n x_i(y_i - b_0 - b_1 x_i) = 0.

Solving these normal equations gives:

\boxed{\hat{\beta}_1 = \frac{S_{xy}}{S_{xx}}, \qquad \hat{\beta}_0 = \bar{y} - \hat{\beta}_1\,\bar{x},}

where S_{xx} = \sum_{i=1}^n (x_i - \bar{x})^2 and \quad S_{xy} = \sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}).

Key insights

• The OLS line always passes through the sample means (\bar{x}, \bar{y}).
• \hat{\beta}_1 is proportional to the sample covariance of x and y.

If we assume GM1–GM4 then Y_i \sim \mathcal{N}(\beta_0 + \beta_1 x_i,\, \sigma^2), \quad i = 1, \ldots, n.

The likelihood function is: \begin{aligned} L(\beta_0, \beta_1, \sigma^2) = \prod_{i=1}^n f_Y(y_i) = (2\pi\sigma^2)^{-\frac{n}{2}} \exp\left(-\frac{1}{2\sigma^2} \sum_{i=1}^n(y_i - \beta_0 - \beta_1 x_i)^2\right). \end{aligned}

The log-likelihood function is: l(\beta_0, \beta_1, \sigma^2) = -\frac{n}{2}\log(2\pi\sigma^2) - \frac{1}{2\sigma^2} \underbrace{\sum_{i=1}^n (y_i - \beta_0 - \beta_1 x_i)^2}_{\text{RSS}}.

Maximizing L with respect to \beta_0 and \beta_1 is equivalent to minimizing the RSS, so: \hat{\beta}_0^{\text{MLE}} = \hat{\beta}_0^{\text{OLS}}, \qquad \hat{\beta}_1^{\text{MLE}} = \hat{\beta}_1^{\text{OLS}}.

• The residuals satisfy the following properties:
  • \sum_{i=1}^n e_i = 0: residuals sum to zero.
  • \sum_{i=1}^n x_i e_i = 0: residuals are uncorrelated with x.
  • \sum_{i=1}^n \hat{y}_i\, e_i = 0: residuals are uncorrelated with fitted values.
• These three identities follow directly from the normal equations.

We can write \hat{\beta}_1 = \sum_{i=1}^n c_i Y_i where c_i = (x_i - \bar{x})/S_{xx} (linear combination).

Since Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i and \mathbb{E}[\varepsilon_i] = 0:

\begin{aligned} \mathbb{E}[\hat{\beta}_1] & = \sum_{i=1}^n c_i \mathbb{E}[Y_i] = \beta_0\underbrace{\sum_{i=1}^n c_i}_{=\frac{n\bar{x}-n\bar{x}}{S_{xx}}=0} + \beta_1 \underbrace{\sum_{i=1}^nc_ix_i}_{=\frac{S_{xx}}{S_{xx}}=1} = \beta_1. \quad \checkmark \end{aligned}

An analogous argument gives \mathbb{E}[\hat{\beta}_0] = \beta_0.

• Both estimators are unbiased under GM1–GM3. Normality (GM4) is not needed.
• On average across many datasets, OLS will return the true parameter values.

Under GM1–GM4, the exact sampling variances are:

\text{Var}(\hat{\beta}_1) = \frac{\sigma^2}{S_{xx}}, \qquad \text{Var}(\hat{\beta}_0) = \sigma^2\!\left(\frac{1}{n} + \frac{\bar{x}^2}{S_{xx}}\right).

What drives precision?

Estimating \sigma^2

Since \sigma^2 is unknown, we estimate it with the residual variance: \hat{\sigma}^2 = \frac{\text{RSS}}{n - 2}.

We divide by n - 2 (not n) because estimating \hat{\beta}_0 and \hat{\beta}_1 costs two degrees of freedom.

Unpacking “BLUE”

• Linear: the estimator is a linear function of the responses Y_1, \ldots, Y_n.
• Unbiased: \mathbb{E}[\hat{\beta}_j] = \beta_j (as shown above).
• Best: among all linear unbiased estimators, OLS has the smallest variance.

The practical message

• We don’t need normality for OLS to be the optimal linear estimator.
• Normality is only needed for exact t-based inference in small samples.
• For large n, the CLT justifies normal-based inference regardless.

Since \hat{\beta}_1 = \sum_i c_i Y_i is a linear combination of independent normals:

\hat{\beta}_1 \sim \mathcal{N}\!\left(\beta_1,\; \frac{\sigma^2}{S_{xx}}\right).

Replacing \sigma with sample standard deviation s_x

Because we estimate \sigma^2 from the same data, we incur extra uncertainty. This leads to:

T = \frac{\hat{\beta}_1 - \beta_1}{s_{\hat{\beta}_1}} \sim t_{n-2}, \qquad s_{\hat{\beta}_1} = \frac{s_x}{\sqrt{S_{xx}}}.

• The degrees of freedom are n - 2 because we estimated two parameters (\beta_0 and \beta_1).
• An analogous result holds for \hat{\beta}_0 with its own standard error.

The most common test is: H_0: \beta_1 = 0 \quad \text{vs.} \quad H_1: \beta_1 \neq 0.

If \beta_1 = 0, then X has no linear association with Y in the population.

Test statistic and p-value

T = \frac{\hat{\beta}_1}{s_{\hat{\beta}_1}} \sim t_{n-2} \quad \text{under } H_0.

p\text{-value} = 2\mathbb{P}(t_{n-2} \geq |T_{\text{obs}}|).

A (1 - \alpha) \times 100\% confidence interval for \beta_j is:

\hat{\beta}_j \;\pm\; t_{\alpha/2,\, n-2} \cdot s_{\hat{\beta}_j}.

Interpretation

• Values of \beta_j outside the CI are inconsistent with the data at the \alpha level.
• If the CI for \beta_1 excludes zero, we reject H_0: \beta_1 = 0 at the same level.
• Reporting a CI is more informative than a p-value alone — it communicates magnitude and uncertainty, not just significance.

Properties of R^2

• 0 \leq R^2 \leq 1.
• R^2 = 0: the model explains nothing (the flat line \hat{y} = \bar{y} fits just as well).
• R^2 = 1: all points lie exactly on the line — perfect fit.
• For SLR only: R^2 = r^2, the square of the Pearson correlation coefficient.

• A high R^2 does not mean the model assumptions are satisfied.
• R^2 always increases when predictors are added — addressed with adjusted R^2 in Lecture 11.

For a new input x^*, the point prediction is: \hat{y}^* = \hat{\beta}_0 + \hat{\beta}_1 x^*.

Two different prediction questions

We may want to estimate:

1. The average response at x^*: \mathbb{E}[Y \mid X = x^*] = \beta_0 + \beta_1 x^*.
2. A single new observation Y^*: Y^* = \beta_0 + \beta_1 x^* + \varepsilon^*.

Both share the same point estimate \hat{y}^*, but they involve fundamentally different uncertainties.

Confidence interval for \mathbb{E}[Y \mid X = x^*] — uncertainty in the mean response:

\hat{y}^* \;\pm\; t_{\alpha/2,\, n-2}\;\hat{\sigma}\sqrt{\frac{1}{n} + \frac{(x^* - \bar{x})^2}{S_{xx}}}.

Prediction interval for a new observation Y^* — uncertainty in a single future value:

\hat{y}^* \;\pm\; t_{\alpha/2,\, n-2}\;\hat{\sigma}\sqrt{1 + \frac{1}{n} + \frac{(x^* - \bar{x})^2}{S_{xx}}}.

Key differences

• The prediction interval carries an extra \hat{\sigma}^2 for the new irreducible error \varepsilon^*.
• Prediction intervals are always wider than confidence intervals.
• Both intervals widen as x^* moves away from \bar{x} — extrapolation is less certain.

✅ What we covered

• Regression as estimation of \mathbb{E}[Y \mid X] — connecting to Lectures 7 & 8.
• The SLR statistical model Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i and the four Gauss-Markov assumptions.
• OLS estimation: deriving \hat{\beta}_0 and \hat{\beta}_1 in closed form by minimising RSS.
• Properties: unbiasedness, sampling variances, and the Gauss-Markov theorem (BLUE).
• Inference: t-tests and confidence intervals for regression coefficients.
• Goodness of fit: SST/SSR/SSE decomposition and the R^2 statistic.
• Prediction: the distinction between CIs for the mean response and prediction intervals.

📅 What’s next?

• Applying SLR end-to-end in Python using a real dataset.
• Fitting a model with statsmodels and reading every line of the regression summary.
• Checking the Gauss-Markov assumptions through residual diagnostic plots.