第 3 章 概率学习Chapter 3 Probabilistic learning

集合 $\mathcal{Y}$ 上的离散概率分布由其概率质量函数（PMF）$p\colon\mathcal{Y}\to[0,1]$ 给定。于是，${\bm{y}}\in\mathcal{Y}$ 的概率定义为A discrete probability distribution over a set $\mathcal{Y}$ is specified by its probability mass function (PMF) $p\colon\mathcal{Y}\to[0,1]$. The probability of ${\bm{y}}\in\mathcal{Y}$ is then defined by

$$\mathbb{P}(Y={\bm{y}})\coloneqq p({\bm{y}}),$$

其中 $Y$ 表示随机变量。当 $Y$ 服从分布 $p$ 时，我们记作 $Y\sim p$（这里略微滥用记号：用同一个字母 $p$ 同时表示分布与 PMF）。于是，$\phi(Y)$ 的期望为（其中 $Y\sim p$，$\phi\colon\mathcal{Y}\to\mathbb{R}^{M}$）where $Y$ denotes a random variable. When $Y$ follows a distribution $p$, we write $Y\sim p$ (with some abuse of notation, we use the same letter $p$ to denote the distribution and the PMF). The expectation of $\phi(Y)$, where $Y\sim p$ and $\phi\colon\mathcal{Y}\to\mathbb{R}^{M}$, is then

$$\mathbb{E}[\phi(Y)]=\sum_{{\bm{y}}\in\mathcal{Y}}p({\bm{y}})\phi({\bm{y}}),$$

其方差（一维变量情形）为its variance (for one-dimensional variables) is

$$\mathbb{V}[\phi(Y)]=\mathbb{E}[(\phi(Y)-\mathbb{E}[\phi(Y)])^{2}]=\sum_{{\bm{y}}\in\mathcal{Y}}p({\bm{y}})(\phi({\bm{y}})-\mathbb{E}[\phi(Y)])^{2}$$

而其众数为and its mode is

$$\argmax_{{\bm{y}}\in\mathcal{Y}}p({\bm{y}}).$$

集合 $\mathcal{Y}$ 上两个离散分布之间的 Kullback-Leibler（KL）散度（亦称相对熵），在相应 PMF 分别为 $p$ 与 $q$ 时，是如下定义的统计“距离”The Kullback-Leibler (KL) divergence (also known as relative entropy) between two discrete distributions over $\mathcal{Y}$, with associated PMFs $p$ and $q$, is the statistical “distance” defined by

$$\mathrm{KL}(p,q)\coloneqq\sum_{{\bm{y}}\in\mathcal{Y}}p({\bm{y}})\log\left(\frac{p({\bm{y}})}{q({\bm{y}})}\right)=\mathbb{E}_{Y\sim p}\log\left(\frac{p(Y)}{q(Y)}\right).$$

集合 $\mathcal{Y}$ 上的连续概率分布由其概率密度函数（PDF）$p\colon\mathcal{Y}\to\mathbb{R}_{+}$ 给定。于是，$\mathcal{A}\subseteq\mathcal{Y}$ 的概率为A continuous probability distribution over $\mathcal{Y}$ is specified by its probability density function (PDF) $p\colon\mathcal{Y}\to\mathbb{R}_{+}$. The probability of $\mathcal{A}\subseteq\mathcal{Y}$ is then

$$\mathbb{P}(Y\in\mathcal{A})=\int_{\mathcal{A}}p(y)dy.$$

期望、方差与 KL 散度的定义均与离散情形类似，只需把 $\sum_{{\bm{y}}\in\mathcal{Y}}$ 换作 $\int_{\mathcal{Y}}$。具体而言，$\phi(Y)$ 的期望为The expectation, variance and KL divergence are defined analogously to the discrete setting, simply replacing $\sum_{{\bm{y}}\in\mathcal{Y}}$ with $\int_{\mathcal{Y}}$. Specifically, the expectation of $\phi(Y)$ is

$$\mathbb{E}[\phi(Y)]=\int_{\mathcal{Y}}p(y)\phi(y)dy,$$

方差为the variance is

$$\mathbb{V}[\phi(Y)]=\mathbb{E}[(\phi(Y)-\mathbb{E}[\phi(Y)])^{2}]=\int_{\mathcal{Y}}p(y)(\phi(y)-\mathbb{E}[\phi(Y)])^{2}dy$$

而 KL 散度为and the KL divergence is

$$\mathrm{KL}(p,q)\coloneqq\int_{\mathcal{Y}}p(y)\log\left(\frac{p(y)}{q(y)}\right)dy=\mathbb{E}_{Y\sim p}\log\left(\frac{p(Y)}{q(Y)}\right).$$

众数则定义为 PDF 的 argmax。The mode is defined as the argmax of the PDF.

当 $\mathcal{Y}=\mathbb{R}$ 时，还可定义累积分布函数（CDF）When $\mathcal{Y}=\mathbb{R}$, we can also define the cumulative distribution function (CDF)

$$\mathbb{P}(Y\leq b)=\int_{-\infty}^{b}p(y)dy.$$

于是，$Y$ 落入半闭区间 $(a,b]$ 内的概率为The probability of $Y$ lying in the semi-closed interval $(a,b]$ is then

$$\mathbb{P}(a<Y\leq b)=\mathbb{P}(Y\leq b)-\mathbb{P}(Y\leq a).$$

我们已经看到，集合 $\mathcal{Y}$ 上的概率分布由 $p({\bm{y}})$ 给定：对离散变量它称为概率质量函数（PMF），对连续变量则称为概率密度函数（PDF）。实际中，生成数据的真实分布 $p$ 未知，我们希望用一个带参数 ${\bm{\lambda}}\in\Lambda$ 的分布 $p_{\bm{\lambda}}$ 来近似它。现给定一组有限的独立同分布观测 ${\bm{y}}_{1},\dots,{\bm{y}}_{N}$，应如何将 ${\bm{\lambda}}\in\Lambda$ 拟合到数据？这可以通过最大化数据的似然来实现，即我们求解We saw that a probability distribution over $\mathcal{Y}$ is specified by $p({\bm{y}})$, which is called the probability mass function (PMF) for discrete variables or the probability density function (PDF) for continuous variables. In practice, the true distribution $p$ generating the data is unknown and we wish to approximate it with a distribution $p_{\bm{\lambda}}$, with parameters ${\bm{\lambda}}\in\Lambda$. Given a finite set of i.i.d. observations ${\bm{y}}_{1},\dots,{\bm{y}}_{N}$, how do we fit ${\bm{\lambda}}\in\Lambda$ to the data? This can be done by maximizing the likelihood of the data, i.e., we seek to solve

$$\widehat{\bm{\lambda}}_{N}\coloneqq\argmax_{{\bm{\lambda}}\in\Lambda}\prod_{i=1}^{N}p_{{\bm{\lambda}}}({\bm{y}}_{i}).$$

这称为最大似然估计（MLE）。由于 $\log$ 函数单调递增，此举等价于最小化负对数似然，即我们有This is known as maximum likelihood estimation (MLE). Because the $\log$ function is monotonically increasing, this is equivalent to minimizing the negative log-likelihood, i.e., we have

$$\widehat{\bm{\lambda}}_{N}=\argmin_{{\bm{\lambda}}\in\Lambda}-\sum_{i=1}^{N}\log p_{{\bm{\lambda}}}({\bm{y}}_{i}).$$

众所周知，MLE 在 Kullback-Leibler 散度意义下是一致的。也就是说，记真实分布为 $p$，并且It is well-known that the MLE is consistent, in the sense of the Kullback-Leibler divergence. That is, denoting the true distribution $p$ and

$${\bm{\lambda}}_{\infty}\coloneqq\argmin_{{\bm{\lambda}}\in\Lambda}\mathrm{KL}(p,p_{\bm{\lambda}})$$

其中，where

$$\mathrm{KL}(p,p_{\bm{\lambda}})=\mathbb{E}_{Y\sim p}\log\left(\frac{p(Y)}{p_{\bm{\lambda}}(Y)}\right),$$

那么，对观测取期望后有 $\widehat{\bm{\lambda}}_{N}\to{\bm{\lambda}}_{\infty}$，当 $N\to\infty$ 时成立。这一结论可通过使用then $\widehat{\bm{\lambda}}_{N}\to{\bm{\lambda}}_{\infty}$ in expectation over the observations, as $N\to\infty$. This can be seen by using

$$\mathrm{KL}(p,p_{\bm{\lambda}})\approx\frac{1}{N}\sum_{i=1}^{N}\log\left(\frac{p({\bm{y}}_{i})}{p_{\bm{\lambda}}({\bm{y}}_{i})}\right)=\frac{1}{N}\sum_{i=1}^{N}\left(\log p({\bm{y}}_{i})-\log p_{\bm{\lambda}}({\bm{y}}_{i})\right)$$

以及大数定律得到。and the law of large numbers.

在机器学习中，很多时候我们希望针对某个输入 ${\bm{x}}\in\mathcal{X}$ 定义条件概率 $\mathbb{P}(Y={\bm{y}}|X={\bm{x}})$，而不是针对某个 ${\bm{y}}\in\mathcal{Y}$ 定义概率 $\mathbb{P}(Y={\bm{y}})$。这可以通过归约为无条件概率分布来实现，Many times in machine learning, instead of a probability $\mathbb{P}(Y={\bm{y}})$ for some ${\bm{y}}\in\mathcal{Y}$, we wish to define a conditional probability $\mathbb{P}(Y={\bm{y}}|X={\bm{x}})$, for some input ${\bm{x}}\in\mathcal{X}$. This can be achieved by reduction to an unconditional probability distribution,

$$\mathbb{P}(Y={\bm{y}}|X={\bm{x}})\coloneqq p_{\bm{\lambda}}({\bm{y}})$$

其中where

$${\bm{\lambda}}\coloneqq f({\bm{x}},{\bm{w}})$$

而 $f$ 是一个带有模型参数 ${\bm{w}}\in\mathcal{W}$ 的模型函数。也就是说，$f$ 并非从 $\mathcal{X}$ 到 $\mathcal{Y}$ 的确定性函数，而是从 $\mathcal{X}$ 到 $\Lambda$ 的函数，其中后者是与输入相关联的输出分布所允许的分布参数集合。and $f$ is a model function with model parameters ${\bm{w}}\in\mathcal{W}$. That is, rather than being a deterministic function from $\mathcal{X}$ to $\mathcal{Y}$, $f$ is a function from $\mathcal{X}$ to $\Lambda$, the set of permissible distribution parameters of the output distribution associated with the input.

我们要强调，${\bm{\lambda}}$ 可以是单个参数，也可以是一组参数的集合。例如，在 伯努利分布中，$\lambda=\pi$；而在一元正态分布中，${\bm{\lambda}}=(\mu,\sigma)$。We emphasize that ${\bm{\lambda}}$ could be a single parameter or a collection of parameters. For instance, in the Bernoulli distribution, $\lambda=\pi$, while in the univariate normal distribution, ${\bm{\lambda}}=(\mu,\sigma)$.

在 3.4 节以及本书其余部分，当 ${\bm{\theta}}$ 是指数族分布的典范参数时，我们也会使用记号 $p_{\bm{\theta}}$ 来代替 $p_{\bm{\lambda}}$。In Section 3.4 and throughout this book, we will also use the notation $p_{\bm{\theta}}$ instead of $p_{\bm{\lambda}}$ when ${\bm{\theta}}$ are the canonical parameters of an exponential family distribution.

这种概率方法的主要优势在于，我们的预测模型比仅仅学习一个从 $\mathcal{X}$ 到 $\mathcal{Y}$ 的函数要丰富得多。现在我们可以获得 $\mathcal{Y}$ 中所有可能结果的整个分布，并计算各种统计量：The main advantage of this probabilistic approach is that our prediction model is much richer than if we just learned a function from $\mathcal{X}$ to $\mathcal{Y}$. We now have access to the whole distribution over possible outcomes in $\mathcal{Y}$ and can compute various statistics:

• •

  概率：$\mathbb{P}(Y={\bm{y}}|X={\bm{x}})$ 或 $\mathbb{P}(Y\in\mathcal{A}|X={\bm{x}})$，Probability: $\mathbb{P}(Y={\bm{y}}|X={\bm{x}})$ or $\mathbb{P}(Y\in\mathcal{A}|X={\bm{x}})$,

• •

  期望：对于某个函数 $\phi$，$\mathbb{E}[\phi(Y)|X={\bm{x}}]$，Expectation: $\mathbb{E}[\phi(Y)|X={\bm{x}}]$ for some function $\phi$,

• •

  方差：$\mathbb{V}[\phi(Y)|X={\bm{x}}]$，Variance: $\mathbb{V}[\phi(Y)|X={\bm{x}}]$,

• •

  众数：$\argmax_{{\bm{y}}\in\mathcal{Y}}p_{\bm{\lambda}}({\bm{y}})$。Mode: $\argmax_{{\bm{y}}\in\mathcal{Y}}p_{\bm{\lambda}}({\bm{y}})$.

现在我们回顾一些概率分布，它们适用于二分类、多分类、回归、多元回归和整数回归。在下文中，为使记号更简洁，我们省略对 ${\bm{x}}$ 的依赖。We now review probability distributions useful for binary classification, multiclass classification, regression, multivariate regression, and integer regression. In the following, to make the notation more lightweight, we omit the dependence on ${\bm{x}}$.

对于二值结果，其中 $\mathcal{Y}=\{0,1\}$，我们可以使用一个伯努利分布，其参数为For binary outcomes, where $\mathcal{Y}=\{0,1\}$, we can use a Bernoulli distribution with parameter

$$\lambda\coloneqq\pi\in[0,1].$$

当随机变量 $Y$ 服从参数为 $\pi$ 的伯努利分布时，我们记作When a random variable $Y$ is distributed according to a Bernoulli distribution with parameter $\pi$, we write

$$Y\sim\mathrm{Bernoulli}(\pi).$$

该分布的概率质量函数为The PMF of this distribution is

$$p_{\pi}(y)\coloneqq\begin{cases}\pi&\text{if }y=1\\ 1-\pi&\text{if }y=0\end{cases}.$$

伯努利分布是单次试验的二项分布。由于 $y\in\{0,1\}$，概率质量函数可改写为The Bernoulli distribution is a binomial distribution with a single trial. Since $y\in\{0,1\}$, the PMF can be rewritten as

$$p_{\pi}(y)=\pi^{y}(1-\pi)^{1-y}.$$

其均值为The mean is

$$\mathbb{E}[Y]=\pi=\mathbb{P}(Y=1)$$

其方差为and the variance is

$$\mathbb{V}[Y]=\pi(1-\pi)=\mathbb{P}(Y=1)\mathbb{P}(Y=0).$$

对于有 $M$ 种可能取值的分类结果，其中 $\mathcal{Y}=[M]$，我们可以使用一个分类分布，其参数为For categorical outcomes with $M$ possible choices, where $\mathcal{Y}=[M]$, we can use a categorical distribution with parameters

$${\bm{\lambda}}\coloneqq{\bm{\pi}}\in\triangle^{M},$$

的 分类分布，其中我们定义概率单纯形where we define the probability simplex

$$\triangle^{M}\coloneqq\{{\bm{\pi}}\in\mathbb{R}_{+}^{M}\colon\langle{\bm{\pi}},\operatorname{\mathbf{1}}\rangle=1\},$$

即有效离散概率分布的集合。当 $Y$ 服从参数为 ${\bm{\pi}}$ 的分类分布时，我们记作i.e., the set of valid discrete probability distributions. When $Y$ follows a categorical distribution with parameter ${\bm{\pi}}$, we write

$$Y\sim\mathrm{Categorical}({\bm{\pi}}).$$

分类分布的概率质量函数为The PMF of the categorical distribution is

$$p_{\bm{\pi}}(y)\coloneqq\langle{\bm{\pi}},\phi(y)\rangle=\pi_{y},$$

其中where

$$\phi(y)\coloneqq{\bm{e}}_{y}$$

是坐标 $y\in[M]$ 的标准基向量。is the standard basis vector for the coordinate $y\in[M]$.

由于 $Y$ 是 分类变量，计算 $Y$ 的期望并无意义，但我们可以计算 $\phi(Y)={\bm{e}}_{Y}$ 的期望，Since $Y$ is a categorical variable, it does not make sense to compute the expectation of $Y$ but we can compute that of $\phi(Y)={\bm{e}}_{Y}$,

$$\mathbb{E}_{Y\sim p_{\bm{\pi}}}[\phi(Y)]={\bm{\pi}}.$$

因此，与 伯努利分布的情形相同，此处均值与概率分布（由向量 ${\bm{\pi}}$ 表示）也是一致的。Therefore, as was also the case for the Bernoulli distribution, the mean and the probability distribution (represented by the vector ${\bm{\pi}}$) are the same in this case.

对于实值结果，其中 $\mathcal{Y}=\mathbb{R}$，可供选择的方案之一是采用具有如下参数的正态分布：For real outcomes, where $\mathcal{Y}=\mathbb{R}$, we can use, among other choices, a normal distribution with parameters

$${\bm{\lambda}}\coloneqq(\mu,\sigma),$$

其中 $\mu\in\mathbb{R}$ 为均值参数，$\sigma\in\mathbb{R}_{+}$ 为标准差参数。若 $Y$ 服从参数为 $(\mu,\sigma)$ 的正态分布，则记作where $\mu\in\mathbb{R}$ is the mean parameter and $\sigma\in\mathbb{R}_{+}$ is the standard deviation parameter. When $Y$ follows a normal distribution with parameters $(\mu,\sigma)$, we write

$$Y\sim\mathrm{Normal}(\mu,\sigma).$$

其概率密度函数为The PDF is

$$p_{\mu,\sigma}(y)\coloneqq\frac{1}{\sigma\sqrt{2\pi}}\exp\left(-\frac{1}{2}\frac{(y-\mu)^{2}}{\sigma^{2}}\right).$$

其期望为The expectation is

$$\mathbb{E}_{Y\sim p_{\mu,\sigma}}[Y]=\mu.$$

概率视角的优点之一在于，我们并不局限于预测均值，还可以计算累积分布函数One advantage of the probabilistic perspective is that we are not limited to predicting the mean. We can also compute the CDF

$$\mathbb{P}(Y\leq y)=\frac{1}{2}\left[1+\mathrm{erf}\left(\frac{y-\mu}{\sigma\sqrt{2}}\right)\right],$$

这里用到了误差函数where we used the error function

$$\operatorname{erf}(z)\coloneqq\frac{2}{\sqrt{\pi}}\int_{0}^{z}e^{-t^{2}}\,\mathrm{d}t.$$

该函数在多数科学计算库中均可调用，例如 SciPy [203]。我们也可将其写作This function is available in most scientific computing libraries, such as SciPy [203]. We can also write

$$\mathbb{P}(Y\leq y)=\Phi\left(\frac{y-\mu}{\sigma}\right)$$

其中where

$$\Phi(z)\coloneqq\frac{1}{2}\left[1+\mathrm{erf}\left(\frac{z}{\sqrt{2}}\right)\right]$$

(3.1)

为标准高斯分布（均值为零、单位方差）的累积分布函数。借助累积分布函数，我们还能方便地得到is the CDF of the standard Gaussian distribution (with zero mean and unit variance). From the CDF, we also easily obtain

$$\mathbb{P}(a<Y\leq b)=\frac{1}{2}\left[\mathrm{erf}\left(\frac{b-\mu}{\sigma\sqrt{2}}\right)-\mathrm{erf}\left(\frac{a-\mu}{\sigma\sqrt{2}}\right)\right].$$

更一般地，对于多变量结果，其中 $\mathcal{Y}=\mathbb{R}^{M}$，我们可以采用如下参数的多元正态分布：More generally, for multivariate outcomes, where $\mathcal{Y}=\mathbb{R}^{M}$, we can use a multivariate normal distribution with parameters

$${\bm{\lambda}}\coloneqq({\bm{\mu}},{\bm{\Sigma}}),$$

其中 ${\bm{\mu}}\in\mathbb{R}^{M}$ 为均值，${\bm{\Sigma}}\in\mathbb{R}^{M\times M}$ 为协方差矩阵。若 $Y$ 服从参数为 $({\bm{\mu}},{\bm{\Sigma}})$ 的多元正态分布，则记作where ${\bm{\mu}}\in\mathbb{R}^{M}$ is the mean and ${\bm{\Sigma}}\in\mathbb{R}^{M\times M}$ is the covariance matrix. When $Y$ follows a multivariate normal distribution with parameters $({\bm{\mu}},{\bm{\Sigma}})$, we write

$$Y\sim\mathrm{Normal}({\bm{\mu}},{\bm{\Sigma}}).$$

概率密度函数为The PDF is

$$p_{{\bm{\mu}},{\bm{\Sigma}}}({\bm{y}})\coloneqq\frac{1}{\sqrt{(2\pi)^{M}|{\bm{\Sigma}}|}}\exp\left(-\frac{1}{2}\langle{\bm{y}}-{\bm{\mu}},{\bm{\Sigma}}^{-1}({\bm{y}}-{\bm{\mu}})\rangle\right).$$

使用对角协方差矩阵，等价于对每个 $Y_{j}$ 分别使用 $M$ 个独立的正态分布，其中 $j\in[M]$。其期望为Using a diagonal covariance matrix is equivalent to using $M$ independent normal distributions for each $Y_{j}$, for $j\in[M]$. The expectation is

$$\mathbb{E}_{Y\sim p_{{\bm{\mu}},{\bm{\Sigma}}}}[Y]={\bm{\mu}}.$$

对于取值为整数的结果，即 $\mathcal{Y}=\mathbb{N}$，我们可以选用均值参数为 $\lambda>0$ 的 Poisson 分布，当然也可作其他选择。当 $Y$ 服从参数为 $\lambda$ 的 Poisson 分布时，我们记作For integer outcomes, where $\mathcal{Y}=\mathbb{N}$, we can use, among other choices, a Poisson distribution with mean parameter $\lambda>0$. When $Y$ follows a Poisson distribution with parameter $\lambda$, we write

$$Y\sim\mathrm{Poisson}(\lambda).$$

其概率质量函数为The PMF is

$$\mathbb{P}(Y=y)=p_{\lambda}(y)\coloneqq\frac{\lambda^{y}\exp(-\lambda)}{y!}.$$

它表示在一段时间区间内发生 $y$ 个事件的概率。当可能事件数量很大、而每个事件都很罕见时，常使用 Poisson 分布。It is the probability of $y$ events occurring in an interval of time. The Poisson distribution is frequently used when there is a large number of possible events, each of which is rare.

其累积分布函数为The CDF is

$$\mathbb{P}(Y\leq y)=\sum_{i=0}^{y}\mathbb{P}(Y=i).$$

Poisson 分布意味着离散指数（方差与均值之比）为 $1$，因为The Poisson distribution implies that the index of dispersion (the ratio between variance and mean) is $1$, since

$$\mathbb{E}[Y]=\mathbb{V}[Y]=\lambda.$$

当这一假设不再适用时，可以使用广义 Poisson 分布 [176]。When this assumption is inappropriate, one can use generalized Poisson distributions [176].

下面讨论如何从输入—输出对 $({\bm{x}}_{1},{\bm{y}}_{1}),\dots,({\bm{x}}_{N},{\bm{y}}_{N})$ 中学习模型参数 ${\bm{w}}\in\mathcal{W}$。We now discuss how to learn the model parameters ${\bm{w}}\in\mathcal{W}$ from input-output pairs $({\bm{x}}_{1},{\bm{y}}_{1}),\dots,({\bm{x}}_{N},{\bm{y}}_{N})$.

指数族是一类概率分布，其 PMF 或 PDF 可以写成如下形式：The exponential family is a class of probability distributions, whose PMF or PDF can be written in the form

$$\displaystyle p_{\bm{\theta}}({\bm{y}})$$	$$\displaystyle=\frac{h({\bm{y}})\exp\left[\langle{\bm{\theta}},\phi({\bm{y}})\rangle\right]}{\exp(A({\bm{\theta}}))}$$
	$$\displaystyle=h({\bm{y}})\exp\left[\langle{\bm{\theta}},\phi({\bm{y}})\rangle-A({\bm{\theta}})\right],$$

其中 ${\bm{\theta}}$ 是分布的自然参数或典范参数。函数 $h$ 称为基测度；函数 $\phi$ 为充分统计量，它承载了关于 ${\bm{y}}$ 的全部信息，并用于将 ${\bm{y}}$ 嵌入向量空间。函数 $A$ 称为对数配分函数或对数归一化函数（详见下文）。我们在 3.3 节中回顾的所有分布均属于指数族。这里我们略微滥用了记号：以 $p_{\bm{\lambda}}$ 表示原始形式的分布，以 $p_{\bm{\theta}}$ 表示指数族形式的分布。正如我们将看到的，由 ${\bm{\theta}}$ 可得 ${\bm{\lambda}}$，反之亦然。下文将说明如何将分布改写为指数族形式。where ${\bm{\theta}}$ are the natural or canonical parameters of the distribution. The function $h$ is known as the base measure. The function $\phi$ is the sufficient statistic: it holds all the information about ${\bm{y}}$ and is used to embed ${\bm{y}}$ in a vector space. The function $A$ is the log-partition or log-normalizer (see below for details). All the distributions we reviewed in Section 3.3 belong to the exponential family. With some abuse of notation, we use $p_{\bm{\lambda}}$ for the distribution in original form and $p_{\bm{\theta}}$ for the distribution in exponential family form. As we will see, we can go from ${\bm{\theta}}$ to ${\bm{\lambda}}$ and vice versa. We illustrate how to rewrite a distribution in exponential family form below.