1 Definition
A family of distributions is part of the exponential family if the probability mass or density function can be written in the form: \[ p(x|\theta) = h(x) \exp\{\eta(\theta) \cdot T(x) - A(\theta)\} \] Or more generally: \[ p(x|\theta) = h(x) \exp\left\{\sum_{j=1}^{k} \eta_j(\theta) T_j(x) - A(\theta)\right\} \] where:
- \(θ\) is the parameter vector
- \(η(θ)\) are the natural parameters (canonical parameters)
- \(T(x)\) are the sufficient statistics
- \(A(θ)\) is the log-partition function (also called the cumulant generating function)
- \(h(x)\) is the base measure
2 Canonical Form
When expressed in natural parameters \(\eta\) instead of \(\theta\), the exponential family takes its simplest form:
\[p(x|\eta) = h(x) \exp\{\eta \cdot T(x) - A(\eta)\}\]
This canonical form reveals the underlying structure and is often easier to work with analytically.
3 Key Properties
3.1 1. Sufficient Statistics
The sufficient statistics T(x) capture all information about the data relevant to parameter estimation. For n i.i.d. observations: \[ \sum_{i=1}^{n} T(x_i) \text{ is sufficient for } \theta \] ### 2. Log-Partition Function
The log-partition function \(A(\eta)\) ensures the distribution integrates to 1: \[ A(\eta) = \log \int h(x) \exp\{\eta \cdot T(x)\} dx \]
Critical property: The moments of \(T(x)\) can be derived from derivatives of \(A(\eta)\): \[ \mathbb{E}[T(x)] = \nabla A(\eta) \] \[ \text{Cov}[T(x)] = \nabla^2 A(\eta) \] This is why \(A(\eta)\) is the cumulant generating function.
3.2 3. Conjugate Priors
Every exponential family distribution has a natural conjugate prior of the form: \[ p(\eta|\alpha, \lambda) \propto \exp\{\eta \cdot \alpha - \lambda A(\eta)\} \] where \(\alpha\) and \(\lambda\) are hyperparameters. Posterior updates remain in the same family.
4 Common Examples
4.1 1. Normal Distribution (univariate)
\[ p(x|\mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left\{-\frac{(x-\mu)^2}{2\sigma^2}\right\} \] Natural parameters: \(\eta_1 = \mu/\sigma^2\), \(\eta_2 = -1/(2\sigma^2)\) Sufficient statistics: \(T_1(x) = x\), \(T_2(x) = x^2\) Log-partition: \(A(\eta) = -\eta_1^2/(4\eta_2) - \frac{1}{2} \log(-2\eta_2)\)
4.2 2. Bernoulli Distribution
\[p(x|p) = p^x(1-p)^{1-x}\]
Natural parameter: \(\eta = \log(p/(1-p))\) Sufficient statistic: \(T(x) = x\) Log-partition: \(A(\eta) = \log(1 + e^\eta)\)
4.3 3. Poisson Distribution
\[p(x|\lambda) = \frac{\lambda^x e^{-\lambda}}{x!}\]
Natural parameter: \(\eta = \log(\lambda)\) Sufficient statistic: \(T(x) = x\) Log-partition: \(A(\eta) = e^\eta\)
4.4 4. Exponential Distribution (scalar parameter)
\[p(x|\beta) = \frac{1}{\beta} e^{-x/\beta}\]
Natural parameter: \(\eta = -1/\beta\) Sufficient statistic: \(T(x) = x\) Log-partition: \(A(\eta) = -\log(-\eta)\)
4.5 5. Beta Distribution
\[p(x|α, β) = \frac{\Gamma(α+β)}{\Gamma(α)\Gamma(β)} x^{α-1}(1-x)^{β-1}\]
Natural parameters: \(\eta_1 = \alpha - 1\), \(\eta_2 = \beta - 1\) Sufficient statistics: \(T_1(x) = \log(x)\), \(T_2(x) = \log(1-x)\)
4.6 6. Multinomial Distribution
Generalization of Bernoulli to k categories.
5 Maximum Likelihood Estimation
For the exponential family, the MLE has a closed form:
\[ \hat{\eta}_{\text{MLE}} = \arg_{\eta} \max_{\theta} \sum_i \eta \cdot T(x_i) - nA(\eta) \]
At the optimum: \[ \mathbb{E}[T(x)] = \frac{1}{n}\sum_i T(x_i) \]
This is a fundamental result: the expected sufficient statistics equal the empirical sufficient statistics.
6 Why the Exponential Family Matters
- Theoretical elegance: Unified framework for diverse distributions
- Computational efficiency: Closed-form solutions for MLEs, conjugate priors, and moments
- Variational inference: Exponential families are the natural choice for variational inference and expectation propagation
- Sufficient Statistics: Enables data reduction without loss of information
- Generative modeling: Foundation for exponential family models, GLMs, and more
7 Connections to Other Concepts
- [[generalized-linear-models]] (GLMs): GLMs assume the response follows an exponential family distribution
- Information Geometry: The exponential family is the natural structure for differential geometry in statistics
- Variational Bayes: Uses exponential families for tractable approximate inference
- Statistical Physics: Exponential families are Gibbs distributions in statistical mechanics
8 Non-Examples
Not all distributions are in the exponential family. Examples include: - Mixture models (not in exponential family form themselves) - Uniform distribution on variable support U(0, θ)—support depends on parameter - Cauchy distribution—moments don’t exist