1 Parametric Point Estimation
Parametric point estimation is the statistical method of using sample data to construct point estimates (single value guesses) of an unknown population parameter (e.g. the population mean).
1.1 Fundamentals
Let \(X\) be an random variable defined on a probability space \((Ω,\mathcal{F},\mathbb{P})\). Recall that a statistic is some function \(T(\boldsymbol{X})\) of the sample \(\boldsymbol{X}:=(X_{1}, \dots, X_{n})\) of realizations of \(X\). Suppose that the distribution function \(F\) of \(X\) depends on a certain number of parameters, and suppose further that the functional form of \(F\) is known except perhaps for a finite number of these parameters. Let \(\boldsymbol{\theta} =(θ_{1},θ_{2},...,θ_{k})\) be the unknown parameter associated with \(F_{\theta}\).
The set of all admissible values of the parameters \(\Theta\) of a distribution function \(F_{\theta}\) is called the parameter space.
Letting \(\psi\) be a real-valued function on \(\Theta\) (for example, a distribution function) we wish to approximate \(\psi(\boldsymbol{\theta})\) given some observations \(\boldsymbol{x}\) of \(\boldsymbol{X}\). Such approximations are known as estimates and they are generated by estimators.
For \(\boldsymbol{X}:=(X_{1}, \dots, X_{n})\sim \mathbb{P}_{\boldsymbol{\theta}}\) for \(\boldsymbol{\theta}\in \Theta\), a statistic \(\delta(\boldsymbol{X})\) is said to be a (point) estimator of \(\psi\) if \(\delta:\mathscr{X}\to \Theta\) where \(\mathscr{X}\) is the space of possible values of \(\boldsymbol{X}\).
The purpose of point estimation is to find an estimator \(\delta\) for the unknown parametric function \(\psi(\boldsymbol{\theta})\) that has desirable properties. The value \(\delta(\boldsymbol{x})\) of \(\delta(\boldsymbol{X})\) for the data \(\boldsymbol{x}\) is called an estimate of \(\psi(\boldsymbol{\theta})\).
1.2 Properties of Estimators
Clearly, there is a very large (often infinite) class of estimators to choose from but it is preferable to choose an estimator \(\delta\) as close to \(\psi(\boldsymbol{\theta})\) as possible. Closeness is given by some measure of the distance between the estimate and \(\psi(\boldsymbol{\theta})\) (both real random variables so typically standard real metric \(d(\delta, \psi):=\lvert \delta(\boldsymbol{X})-\psi(\boldsymbol{\theta}) \rvert\)) in probability. The most common error measures are:
- \(\mathbb{P}_{\boldsymbol{\theta}}(\lvert \delta(\boldsymbol{X})-\psi(\boldsymbol{\theta}) \rvert<\varepsilon)\) for some \(\varepsilon >0\); and
- \(\mathbb{E}_{\boldsymbol{\theta}}[\lvert \delta(\boldsymbol{X})-\psi(\boldsymbol{\theta}) \rvert^r]\) for some \(r>0\),
where we wish \((1)\) to be large and \((2)\) to be small. We define the important special case of \((2)\) given for \(r=2\).
The mean square error between an estimator \(\delta(\boldsymbol{X})\) and function \(\psi(\boldsymbol{\theta})\) is given by \[ \text{MSE}_{\boldsymbol{\theta}}(\delta)=\mathbb{E}_{\boldsymbol{\theta}}[(\delta(\boldsymbol{X})-\psi(\boldsymbol{\theta}))^2]. \]
Among all estimators for \(\psi\) we would like to choose one \(\delta_{0}\) such that \(\forall \varepsilon>0\) \[ \mathbb{P}_{\boldsymbol{\theta}}(\lvert \delta_{0}-\psi \rvert <\varepsilon )\geq \mathbb{P}_{\boldsymbol{\theta}}(\lvert \delta -\psi \rvert <\varepsilon);\quad \forall \delta, \] and such that \[ MSE_{\boldsymbol{\theta}}(\delta_{0})\leq MSE_{\boldsymbol{\theta}}(\delta), \] however estimators satisfying both generally don’t exist.
1.2.1 Bias
An important property to consider when selecting estimators is biasedness which is a measure of the systematic error (i.e average overestimation / underestimation).
For estimator \(\delta\) of \(\psi\) the bias of \(\delta\) is given by \[ b(\delta,\psi)=\mathbb{E}_{\boldsymbol{\theta}}[\delta(\boldsymbol{X})]-\psi(\boldsymbol{\theta}). \]
We are often interested in unbiased estimators, that is estimators \(\delta\) where \[ \mathbb{E}_{\boldsymbol{\theta}}[\delta(\boldsymbol{X})]=\psi(\boldsymbol{\theta}); \quad \forall \boldsymbol{\theta}\in \Theta. \] ### Consistency
Next we consider some large sample properties of estimators. As the sample size increases \(n \to \infty\) the data \(\boldsymbol{x}\) are essentially the whole population and we should expect \(\delta\) to approach \(\psi\) in some sense.
A sequence of point estimators \(\{T_{n}(\boldsymbol{X})\}\) is consistent for \(\psi\) if for each fixed \(\boldsymbol{\theta}\in \Theta\) \[ T_{n}\stackrel{\mathbb{P}}\to \psi\quad \text{as}\quad n \to \infty. \]
The following theorem is helpful for determining whether a sequence of estimators is consistent.
If \(T_{n}\) is a sequence of estimators such that \(\mathbb{E}T_{n}\to \psi\) and \(\text{Var}(T_{n})\to 0\) as \(n \to \infty\) then \(T_{n}\) is consistent for \(\psi(\boldsymbol{\theta})\).
Proof: We can use the Chebyshev Inequality to see that \[ \begin{align} \mathbb{P}(\lvert T_{n}-\psi \rvert >\varepsilon) & \leq \frac{1}{\varepsilon^2}\mathbb{E}[(T_{n}-\mathbb{E}T_{n}+\mathbb{E}T_{n}-\psi)^2] \\& =\frac{1}{\varepsilon^2}(\text{Var}(T_{n})+(\mathbb{E}T_{n}-\psi)^2)\to 0 ~~\text{as}~~n \to \infty,\end{align} \] completing the proof. \(\square\)
Some other large sample properties of estimators are asymptotic unbiasedness, asymptotic normality, and asymptotic efficiency. A sequence of estimators \(\{ T_{n} \}\) is asymptotically unbiased for \(\psi\) if \[ \lim_{ n \to \infty } \mathbb{E}_{\theta}[T_{n}]=\psi;\quad \forall \boldsymbol{\theta}. \] A consistent sequence of estimators \(\{ T_{n} \}\) is said to be consistent asymptotically normal (CAN) for \(\psi\) if \(T_{n}\sim \text{AN}(\psi, \frac{v(\boldsymbol{\theta})}{n})\) for all \(\boldsymbol{\theta}\). If \(v(\theta)=\frac{1}{I(\theta)}\) where \(I\) is the [[fisher-information]] then \(\{ T_{n} \}\) known as a best asymptotically normal (BAN) estimator.
1.2.2 Sufficiency
To gain an intuitive idea of sufficiency consider the role of a statistician. Raw data might be very large and difficult to store, visualize and interpret. Therefore the statistician would like to condense the data by computing some statistics that lose no information upon which she may base her analysis.
For sample \(\boldsymbol{X}\) from \(F:=\{F_{\theta} : \theta \in \Theta\}\) statistic \(T(\boldsymbol{X})\) is sufficient for \(\theta\) (or for family \(F\)) if and only if the conditional distribution of \(X\) given \(T=t\) does not depend on \(\theta\) almost surely.
Example: Let \(X_{1}, X_{2}\) be i.i.d. \(\mathcal{P}(\lambda)\) random variables and consider the statistic \(T=X_{1}+2X_{2}\). We show that this statistic is not sufficient. From the definition of conditional probability we have \[ \begin{align} \mathbb{P}(X_{1}=0, X_{2}=1 |X_{1}+2X_{2}=2) & = \frac{\mathbb{P}(X_{1}=0, X_{2}=1)}{\mathbb{P}(X_{1}+2X_{2}=2)} \\ & = \frac{e^{-\lambda}\lambda e^{-\lambda}}{\mathbb{P}(X_{1}=0, X_{2}=1)+\mathbb{P}(X_{2}=2, X_{2}=0)} \\ & = \frac{\lambda e^{-2\lambda}}{\lambda e^{-2\lambda}+\left( \frac{\lambda^2}{2} \right)e^{-2\lambda}} \\ & =\frac{1}{1+\frac{\lambda}{2}}, \end{align} \] hence \(X_{1}+2X_{2}\) is not sufficient.
The definition of sufficiency is not constructive since it requires we first guess a statistic \(T\) before checking for sufficiency through a time consuming computation. The following important theorem gives a criterion for determining sufficient statistics.
For random variables \(\boldsymbol{X}:=(X_{1}, \dots, X_{n})\) with distribution \(f_{\theta}(\boldsymbol{x})\), \(\boldsymbol{\theta}\in \Theta\), the statistic \(T(\boldsymbol{X})\) is sufficient for \(\theta\) if and only if we can write \[ f_{\theta}(\boldsymbol{x})=h(\boldsymbol{x})g_{\boldsymbol{\theta}}(T(\boldsymbol{x})), \] where \(h\) is a nonnegative function of \(\boldsymbol{x}\) only and does not depend on \(\boldsymbol{\theta}\), and \(g_{\theta}\) is a nonnegative nonconstant function of \(\boldsymbol{\theta}\) and \(T(\boldsymbol{x})\) only (where \(T\) and \(\boldsymbol{\theta}\) may be multidimensional).
For a proof of this theorem please be directed to Halmos and Savage 41 or to Lehmann 64, pp. 53–56.
Example:
If \(T\) is sufficient for \(\boldsymbol{\theta}\), any one-to-one function of \(T\) is also sufficient.
Proof: If \(U = k(T)\) is a one-to-one function then \(t=k^{-1}(u)\) and we can write \[ f_{\boldsymbol{\theta}}(\boldsymbol{x})=g_{\boldsymbol{\theta}}(t_{1})h(\boldsymbol{x})=g_{\boldsymbol{\theta}}(k^{-1}(u))h(\boldsymbol{x})=g_{\boldsymbol{\theta}}^*(u)h(\boldsymbol{x}). \] If \(T_{1}, T_{2}\) are two distinct sufficient statistics, then \[ f_{\boldsymbol{\theta}}(\boldsymbol{x})=g_{\boldsymbol{\theta}}(t_{1})h_{1}(\boldsymbol{ x})=g_{\boldsymbol{\theta}}(t_{2})h_{2}(\boldsymbol{x}), \] and it follows that \(T_1\) is a function of \(T_{2}\).
1.2.3 Complete Statistics
The concept of sufficiency is frequently used with the concept of completeness.
We say that the probability distribution family \(\{ f_{\boldsymbol{\theta}}, \boldsymbol{\theta} \in \Theta \}\) is complete if \[ \mathbb{E}_{\boldsymbol{\theta}}[g(X)]=0;~~ \forall \theta \in \Theta, \] which implies that \[ \mathbb{P}_{\theta}(g(X)=0)=1; ~~\forall \theta \in \Theta. \]
A statistic \(T(X)\) is said to be complete if the family of distributions of \(T\) is complete.
Example:
The following result covers the \(k\)-parameter exponential family of distributions.
Let \(\{ f_{\theta}:\theta \in \Theta \}\) be a \(k\)-parameter exponential family given by \[ f_{\theta}(x)=\exp\left( \sum_{j=1}^{k}Q_{j}(\theta)T_{j}(x)+D(\theta)+S(x) \right), \] where \(\theta:=(\theta_{1}, \dots, \theta_{k})\in \Theta\), an interval in \(\mathbb{R}_{k}\), \(T_{1}, \dots, T_{K}\) and \(S\) are defined on \(\mathbb{R}_{n}\), \(T=(T_{1}, \dots, T_{k})\), \(x=(x_{1}, \dots, x_{n})\) and \(k \leq n\). Let \(Q=(Q_{1}, \dots, Q_{k})\) and suppose that the range of \(Q\) contains an open set in \(\mathbb{R}_{k}\). Then \[ T:=(T_{1}(X), \dots, T_{k}(X)), \] is a complete sufficient statistic.
*For a complete proof in a general setting we refer the reader to Lehmann 64, pp. 142–143.**Example:*
The following result covers the \(k\)-parameter exponential family of distributions.
Let \(\{ f_{\theta}:\theta \in \Theta \}\) be a \(k\)-parameter exponential family given by \[ f_{\theta}(x)=\exp\left( \sum_{j=1}^{k}Q_{j}(\theta)T_{j}(x)+D(\theta)+S(x) \right), \] where \(\theta:=(\theta_{1}, \dots, \theta_{k})\in \Theta\), an interval in \(\mathbb{R}_{k}\), \(T_{1}, \dots, T_{K}\) and \(S\) are defined on \(\mathbb{R}_{n}\), \(T=(T_{1}, \dots, T_{k})\), \(x=(x_{1}, \dots, x_{n})\) and \(k \leq n\). Let \(Q=(Q_{1}, \dots, Q_{k})\) and suppose that the range of \(Q\) contains an open set in \(\mathbb{R}_{k}\). Then \[ T:=(T_{1}(X), \dots, T_{k}(X)), \] is a complete sufficient statistic.
For a complete proof in a general setting we refer the reader to Lehmann 64, pp. 142–143.
A given family of probability distributions that admits a nontrivial sufficient statistic usually admits several sufficient statistics. We would like to be able to choose the sufficient statistic that results in the greatest reduction of data collection, introducing the notion of a minimal sufficient statistic. For this purpose it is convenient to Example:
The following result covers the \(k\)-parameter exponential family of distributions.
Let \(\{ f_{\theta}:\theta \in \Theta \}\) be a \(k\)-parameter exponential family given by \[ f_{\theta}(x)=\exp\left( \sum_{j=1}^{k}Q_{j}(\theta)T_{j}(x)+D(\theta)+S(x) \right), \] where \(\theta:=(\theta_{1}, \dots, \theta_{k})\in \Theta\), an interval in \(\mathbb{R}_{k}\), \(T_{1}, \dots, T_{K}\) and \(S\) are defined on \(\mathbb{R}_{n}\), \(T=(T_{1}, \dots, T_{k})\), \(x=(x_{1}, \dots, x_{n})\) and \(k \leq n\). Let \(Q=(Q_{1}, \dots, Q_{k})\) and suppose that the range of \(Q\) contains an open set in \(\mathbb{R}_{k}\). Then \[ T:=(T_{1}(X), \dots, T_{k}(X)), \] is a complete sufficient statistic.
For a complete proof in a general setting we refer the reader to Lehmann 64, pp. 142–143.
A given family of probability distributions that admits a nontrivial sufficient statistic usually admits several sufficient statistics. We would like to be able to choose the sufficient statistic that results in the greatest reduction of data collection, introducing the notion of a minimal sufficient statistic. For this purpose it is convenient to