Linear regression is a statistical model which estimates the linear relationship between a scalar response (dependent) variable and one or more explanatory (independent) variables. The case of one explanatory variable is called simple linear regression and for more than one is called multiple linear regressionwhich is distinct from multivariate linear regression where multiple correlated dependent variables are predicted.
In linear regression the relationships are modelled using linear predictor functions (known as linear models) whose unknown model parameters are estimated from the data. Most commonly the conditional mean of the response given the values of the predictors is assume to be an affine function of those values. Like all forms of regression analysis, linear regression focusses on the conditional probability distribution of the response given the values of the predictors, rather than on the joint probability distribution of all of these variables, which is in the domain of [[multivariate-analysis|multivariate analysis]]. #### Formulation
Given a data set \(\{y_{i}, x_{i1}, ..., x_{ip}\}_{i=1}^{n}\) of \(n\) statistical units, a linear regression model assumes that the relationship between the response \(y\) and the vector of covariates \(x\) is linear. The relationship is modelled through an error variable \(\epsilon\) which adds noise to the relationship.
The model takes the form \[ y_i=\beta_0+\beta_{1}x_{i1}+...+\beta_{p}x_{ip}+\epsilon_i=\vec{x}^{T}_{i}\vec\beta+\epsilon_{i},\quad i=1, ..., n \] Written in matrix form \[ \boldsymbol{y}=\boldsymbol{X\beta}+\boldsymbol{\epsilon} \] where: \[ \boldsymbol{y}=\begin{bmatrix}y_{1}\\\vdots\\y_{n}\end{bmatrix},\quad \boldsymbol{X}=\begin{bmatrix}1&x_{11}&\cdots&x_{1p}\\1&x_{21}&\cdots&x_{2p}\\\vdots&\vdots&\ddots&\vdots\\1&x_{n1}&\cdots&x_{np}\end{bmatrix},\quad \boldsymbol{\beta}=\begin{bmatrix}\beta_0\\\beta_1\\\vdots\\\beta_p\end{bmatrix}, \quad\boldsymbol{\epsilon} = \begin{bmatrix}\epsilon_1\\\vdots\\\epsilon_n\end{bmatrix}. \] #### Model Assumptions
- Weak Exogeneity
- Linearity
- Constant Variance
- Independence of Errors
- Lack of Perfect Multicollinearity