第 14 章 通过积分进行平滑Chapter 14 Smoothing by integration

两个函数 $f$ 与 $g$ 之间的卷积会生成另一个函数，记为 $f\ast g$。其定义为The convolution between two functions $f$ and $g$ produces another function, denoted $f\ast g$. It is defined by

$$(f\ast g)(\mu)\coloneqq\int_{-\infty}^{\infty}f(u)g(\mu-u)\,du,$$

(14.1)

这里假设积分是良定义的。因此，它是将 $f$ 与 $g$ 的乘积，在将 $g$ 关于 $y$ 轴作反射并平移之后进行积分所得的结果。这一运算可以视为滑动平均的推广。利用变量替换 $u\coloneqq\mu+z$，我们也可以将其写成assuming that the integral is well-defined. It is therefore the integral of the product of $f$ and $g$ after $g$ is reflected about the $y$-axis and shifted. It can be seen as a generalization of the moving average. Using the change of variable $u\coloneqq\mu+z$, we can also write

$$\displaystyle(f\ast g)(\mu)$$	$$\displaystyle=\int_{-\infty}^{\infty}f(\mu+z)g(-z)\,dz$$	(14.2)
	$$\displaystyle=\int_{\infty}^{-\infty}f(\mu-v)g(v)\,(-dv)$$
	$$\displaystyle=\int_{-\infty}^{\infty}g(v)f(\mu-v)\,dv$$
	$$\displaystyle=(g\ast f)(\mu).$$

因此，卷积算子是可交换的。The convolution operator is therefore commutative.

卷积经常与核函数 $\kappa$ 一起使用，以构造 $f$ 的光滑近似 $f\ast\kappa$。最常用的核函数是宽度为 $\sigma$ 的高斯核，其定义为The convolution is frequently used together with a kernel $\kappa$ to create a smooth approximation $f\ast\kappa$ of $f$. The most frequently used kernel is the Gaussian kernel with width $\sigma$, defined by

$$\kappa_{\sigma}(z)\coloneqq\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{1}{2}(\frac{z}{\sigma})^{2}}.$$

这是均值为零、方差为 $\sigma^{2}$ 的正态分布的概率密度函数（PDF）。项 $\frac{1}{\sqrt{2\pi}\sigma}$ 是归一化常数，它确保该核函数关于 $\sigma$ 的求和（积分）为 $1$。因此，我们说 $\kappa_{\sigma}$ 是归一化核。This is the probability density function (PDF) of the normal distribution with zero mean and variance $\sigma^{2}$. The term $\frac{1}{\sqrt{2\pi}\sigma}$ is a normalization constant, ensuring that the kernel sums to $1$ for all $\sigma$. We therefore say that $\kappa_{\sigma}$ is a normalized kernel.

很多时候，我们所处理的函数，其卷积并无解析形式。此时，可以在数值网格上采用离散卷积。对于定义在 $\mathbb{Z}$ 上的两个函数 $f$ 与 $g$，离散卷积定义为Many times, we work with functions whose convolution does not have an analytical form. In these cases, we can use a discrete convolution on a grid of values. For two functions $f$ and $g$ defined over $\mathbb{Z}$, the discrete convolution is defined by

$$(f\ast g)[i]\coloneqq\sum_{j=-\infty}^{\infty}f[j]g[i-j].$$

与连续情形相同，离散卷积也是可交换的，即As for its continuous counterpart, the discrete convolution is commutative, namely,

$$(f\ast g)[i]=\sum_{j=-\infty}^{\infty}f[i-j]g[j]=(g\ast f)[i].$$

当 $g$ 在集合 $S\coloneqq\{-M,-M+1,\dots,0,\dots,M-1,M\}$ 上具有有限支撑集，即对任意 $i\not\in S$ 都有 $g[i]=0$ 时，便可将求和限制在有限范围内，即When $g$ has finite support over the set $S\coloneqq\{-M,-M+1,\dots,0,\dots,M-1,M\}$, meaning that $g[i]=0$ for all $i\not\in S$, a finite summation may be used instead, i.e.,

$$(f\ast g)[i]=\sum_{j=-M}^{M}f[i-j]g[j]=(g\ast f)[i].$$

在实践中，离散信号 $f\colon\mathbb{Z}\to\mathbb{R}$ 与连续核函数 $\kappa\colon\mathbb{R}\to\mathbb{R}$ 之间的卷积，是通过对核函数做离散化来实现的。最简单的做法之一，是在一段区间上取采样点，在这些点上对核函数求值，再将所得的值重新归一化，使采样后的核函数之和为 $1$。图 14.1 以高斯核为例展示了这一做法。由于高斯核按指数速度衰减，我们可以围绕 $0$ 选取一个较窄的区间。关于高斯核其他可能离散化方式的综述，见 [75]。In practice, convolution between a discrete signal $f\colon\mathbb{Z}\to\mathbb{R}$ and a continuous kernel $\kappa\colon\mathbb{R}\to\mathbb{R}$ is implemented by discretizing the kernel. One of the simplest approaches consists in sampling points on an interval, evaluating the kernel at these points and renormalizing the obtained values, so that the sampled kernel sums to $1$. This is illustrated with the Gaussian kernel in Fig. 14.1. Since the Gaussian kernel decays exponentially fast, we can choose a small interval around $0$. For a survey of other possible discretizations of the Gaussian kernel, see [75].

值得注意的是，只要这两个函数及其导数均可积，卷积的导数便满足Remarkably, provided that the two functions are integrable with integrable derivatives, the derivative of the convolution satisfies

$$(f\ast g)^{\prime}=(f^{\prime}\ast g)=(f\ast g^{\prime}),$$

这一等式只需在卷积的定义中交换求导与积分的次序即可得到。此外，还有如下命题。which simply stems from switching derivative and integral in the definition of the convolution. Moreover, we have the following proposition.

至此，我们讨论的都是两个一元函数的卷积。该定义可以自然地推广到多元函数 $f\colon\mathbb{R}^{M}\to\mathbb{R}$ 与 $g\colon\mathbb{R}^{M}\to\mathbb{R}$，即So far, we studied the convolution of one-dimensional functions. The definition can be naturally extended to multidimensional functions $f\colon\mathbb{R}^{M}\to\mathbb{R}$ and $g\colon\mathbb{R}^{M}\to\mathbb{R}$ as

$$(f\ast g)({\bm{\mu}})\coloneqq\int_{\mathbb{R}^{M}}f({\bm{u}})g({\bm{\mu}}-{\bm{u}})\,d{\bm{u}},$$

这里同样假设积分存在。通常情况下，所用的高斯核其协方差矩阵为对角矩阵assuming again that the integral exists. Typically, a Gaussian kernel with diagonal covariance matrix is used

$$\displaystyle\kappa_{\sigma}({\bm{z}})$$

$$\displaystyle\coloneqq\prod_{j=1}^{M}\frac{1}{\sqrt{2\pi}\sigma_{j}}e^{-\frac{1}{2}(\frac{z_{j}}{\sigma_{j}})^{2}}=\frac{1}{\sqrt{2\pi}^{M}\sigma^{M}}e^{-\frac{1}{2}\frac{\|{\bm{z}}\|^{2}_{2}}{\sigma^{2}}},$$

(14.3)

其中第二个等式中假定了 $\sigma_{1}=\dots=\sigma_{M}$。在图像处理的语境下（此时 $M=2$），该积分借助离散卷积来近似，所得结果称为高斯模糊。where, in the second equality, we assumed $\sigma_{1}=\dots=\sigma_{M}$. In an image processing context, where $M=2$, it is approximated using a discrete convolution and it is called a Gaussian blur.

我们在 13.1 节中研究过的下确卷积，其形式为The infimal convolution we studied in Section 13.1 takes the form

$$(f\square g)({\bm{\mu}})\coloneqq\inf_{{\bm{u}}\in\mathbb{R}^{M}}f({\bm{u}})+g({\bm{\mu}}-{\bm{u}}).$$

作为对照，经典的卷积则取如下形式In comparison, the classical convolution takes the form

$$(F\ast G)({\bm{\mu}})\coloneqq\int_{\mathbb{R}^{M}}F({\bm{u}})G({\bm{\mu}}-{\bm{u}})\,d{\bm{u}}.$$

这两种卷积显然相互关联。下确卷积取的是下确界，并把 $f$ 与 $g$ 相加：它采用的是最小—加代数。经典卷积则执行积分，并将 $F$ 与 $G$ 相乘：它采用的是和—积代数。The two forms of convolution are clearly related. Infimal convolution performs an infimum and uses the sum of $f$ and $g$: it uses a min-plus algebra. Classical convolution performs an integral and uses the product of $F$ and $G$: it uses a sum-product algebra.

若在下确卷积的定义中以下确界换成软最小值，下确卷积与经典卷积之间的联系便会更加清晰。The link between the infimal convolution and the classical convolution can be further elucidated if we replace the infimum with a soft minimum in the definition of the infimal convolution.

我们便得到下确卷积，即 $\varepsilon\to 0$。We recover the infimal convolution as $\varepsilon\to 0$.

我们在 13.1.2 节中看到，$f$ 与 $R(z)=\frac{1}{2}z^{2}$ 之间的下确卷积即为 Moreau 包络，We saw in Section 13.1.2 that the infimal convolution between $f$ and $R(z)=\frac{1}{2}z^{2}$ is the Moreau envelope,

$$M_{f}({\bm{\mu}})\coloneqq(f\square R)({\bm{\mu}})=\inf_{{\bm{u}}\in\mathbb{R}^{M}}f({\bm{u}})+\frac{1}{2}\|{\bm{\mu}}-{\bm{u}}\|^{2}_{2}.$$

将下确卷积替换为软下确卷积，便可定义“软”Moreau 包络，Replacing the infimal convolution with a soft infimal convolution, we can define the “soft” Moreau envelope,

$$M_{f}^{\varepsilon}({\bm{\mu}})\coloneqq(f\square_{\varepsilon}R)({\bm{\mu}})=\underset{{\bm{u}}\in\mathbb{R}^{M}}{\mathrm{softmin}_{\varepsilon}} \ f({\bm{u}})+\frac{1}{2}\|{\bm{\mu}}-{\bm{u}}\|^{2}_{2}.$$

需要强调，这一运算与 $f$ 同高斯核函数的卷积并不相同。事实上，有We emphasize that this operation is not the same as the convolution of $f$ with a Gaussian kernel. Indeed, we have

$$\displaystyle M_{f}^{\varepsilon}({\bm{\mu}})=-\varepsilon\log\int_{\mathbb{R}^{M}}\exp\left((-f({\bm{u}})-\frac{1}{2}\|{\bm{\mu}}-{\bm{u}}\|_{2}^{2})/\varepsilon\right)d{\bm{u}}.$$

而while

$$\displaystyle(f\ast\kappa_{\sigma})({\bm{\mu}})\coloneqq\int_{\mathbb{R}^{M}}f({\bm{u}})\kappa_{\sigma}({\bm{\mu}}-{\bm{u}})\,d{\bm{u}},$$

其中 $\kappa_{\sigma}$ 例如由式 14.3 定义。where $\kappa_{\sigma}$ is for instance defined in Eq. 14.3.

我们已看到，Moreau 包络是光滑函数。那么，使用软 Moreau 包络究竟能带来什么好处？好处可能在于计算：软 Moreau 包络可用离散卷积来近似。We saw that the Moreau envelope is a smooth function. One may therefore ask what do we gain from using a soft Moreau envelope. The benefit can be computational, as the latter can be approximated using a discrete convolution.

现在考虑卷积Now, consider the convolution

$$h(t)\coloneqq(f\ast g)(t).$$

若定义如下三个变换If we define the three transformations

$$\displaystyle F$$

$$\displaystyle\coloneqq\mathcal{F}\{f\},\ G\coloneqq\mathcal{F}\{g\},\ H\coloneqq\mathcal{F}\{h\},$$

卷积定理指出the convolution theorem states that

$$H(s)=F(s)\cdot G(s),\quad s\in\mathbb{R}.$$

换一种写法，即有Written differently, we have

$$\mathcal{F}\{f\ast g\}=\mathcal{F}\{f\}\cdot\mathcal{F}\{g\}.$$

用文字来说，在 Fourier 域中，卷积运算变为乘法。反过来，In words, in the Fourier domain, the convolution operation becomes a multiplication. Conversely,

$$h(t)=(f*g)(t)=\mathcal{F}^{-1}\{F\cdot G\}(t),\quad t\in\mathbb{R}.$$

若将 Fourier 变换替换为 Laplace 变换或双侧（双边）Laplace 变换，卷积定理同样成立。The convolution theorem also holds if we replace the Fourier transform with the Laplace transform or with the two-sided (bilateral) Laplace transform.

在 13.2 节中，我们研究了另一种函数变换：凸共轭，又称 Legendre-Fenchel 变换。我们在表 14.1 中回顾了这些变换之间的类比关系。特别地，……的对应项In Section 13.2, we studied another function transformation: the convex conjugate, also known as Legendre-Fenchel transform. We recap the analogies between these transforms in Table 14.1. In particular, the counterpart of

$$\mathcal{F}\{f\ast g\}=\mathcal{F}\{f\}\cdot\mathcal{F}\{g\}$$

对于下确卷积，与之对应的则是for the infimal convolution is

$$(f\square g)^{*}=f^{*}+g^{*}.$$

用文字来说，Legendre-Fenchel 变换之于下确卷积，恰如 Fourier 变换之于卷积。In words, the Legendre-Fenchel transform is to the infimal convolution what the Fourier transform is to the convolution.

我们在 13.2 节中看到，函数 $f\colon\mathbb{R}^{M}\to\mathbb{R}$ 的 Legendre-Fenchel 变换（凸共轭）为We saw in Section 13.2 that the Legendre-Fenchel transform (convex conjugate) of a function $f\colon\mathbb{R}^{M}\to\mathbb{R}$ is

$$f^{*}({\bm{v}})\coloneqq\max_{{\bm{u}}\in\mathbb{R}^{M}} \ \langle{\bm{u}},{\bm{v}}\rangle-f({\bm{u}}).$$

必要时，我们可以在 $f$ 的定义中加入一个指示函数来支持约束。该共轭可以通过 log-sum-exp 平滑化，后者起着软最大值的作用（13.5 节）。If necessary, we can support constraints by including an indicator function in the definition of $f$. The conjugate can be smoothed out using a log-sum-exp, which plays the role of a soft maximum (Section 13.5).

在极限 $\varepsilon\to 0$ 下，我们恢复凸共轭。In the limit $\varepsilon\to 0$, we recover the convex conjugate.

为说明卷积的求导，我们展示如何对平滑的 ReLU 求导。To illustrate differentiation of the convolution, we show how to differentiate the smoothed ReLU.

平滑函数 $f:\mathbb{R}^{M}\rightarrow\mathbb{R}$ 的一个相当直观的做法，是让输入 ${\bm{\mu}}$ 受到服从某个噪声分布 $p$ 的加性噪声 $Z\sim p$ 的扰动，再对其取值取平均。由此定义替代函数A rather intuitive approach to smooth a function $f:\mathbb{R}^{M}\rightarrow\mathbb{R}$ is to average its values on an input ${\bm{\mu}}$, perturbed by some additive noise $Z\sim p$, for some noise distribution $p$. This defines the surrogate

$$f_{\sigma}({\bm{\mu}})\coloneqq\mathbb{E}_{Z\sim p}[f({\bm{\mu}}+\sigma Z)].$$

参数 $\sigma$ 控制扰动强度：当 $\sigma\rightarrow 0$ 时，我们自然回到 $f$。若定义变换（变量替换），则可得到等价的视角The parameter $\sigma$ controls the perturbation strength: as $\sigma\rightarrow 0$, we naturally recover $f$. An equivalent viewpoint is obtained by defining the transformation (change of variables)

$$U\coloneqq{\bm{\mu}}+\sigma Z.$$

于是有We then have

$$U\sim p_{{\bm{\mu}},\sigma},$$

其中 $p_{{\bm{\mu}},\sigma}$ 是由噪声分布 $p$ 生成的位置族分布，即 $Z$ 经该变换得到的前推分布（见 12.4.4 节）。在此记号下，初始噪声分布 $p$ 就简单地是 $p=p_{{\mathbf{0}},1}$。由此，扰动后的函数可从这两个视角表示为where $p_{{\bm{\mu}},\sigma}$ is the location-family distribution generated by the noise distribution $p$. It is the pushforward distribution of $Z$ through the transformation (see Section 12.4.4). In this notation, the initial noise distribution $p$ is then simply $p=p_{{\mathbf{0}},1}$. The perturbed function can then be expressed from these two perspectives as

$$\displaystyle f_{\sigma}({\bm{\mu}})$$	$$\displaystyle=\mathbb{E}_{Z\sim p_{{\mathbf{0}},1}}[f({\bm{\mu}}+\sigma\cdot Z)]$$
	$$\displaystyle=\mathbb{E}_{U\sim p_{{\bm{\mu}},\sigma}}[f(U)].$$	(14.4)

将期望写成概率密度函数的积分，便自然得到前文所述通过卷积实现的平滑Writing the expectation as the integral of a p.d.f, we naturally recover the smoothing by convolution presented earlier,

$$\displaystyle f_{\sigma}({\bm{\mu}})$$	$$\displaystyle=\int f({\bm{\mu}}+\sigma{\bm{z}})p_{{\mathbf{0}},1}({\bm{z}})d{\bm{z}}$$
	$$\displaystyle=f\ast\kappa_{\sigma}({\bm{\mu}}),$$

其中我们定义了核函数where we defined the kernel

$$\kappa_{\sigma}({\bm{z}})\coloneqq p_{{\mathbf{0}},\sigma}(-{\bm{z}}).$$

在下文中，我们假设噪声分布可分解为In the sequel, we assume that the noise distribution decomposes as

$$p_{{\mathbf{0}},1}({\bm{z}})\coloneqq\exp(-\nu({\bm{z}}))/C,$$

其中 $\nu({\bm{z}})$ 是噪声分布的对数密度，$C$ 是归一化常数。例如，取 $\nu({\bm{z}})=\frac{1}{2}\|{\bm{z}}\|^{2}_{2}$、$C=\sqrt{2\pi}^{M}$，即得到具有对角协方差矩阵的高斯分布及相应的高斯核。where $\nu({\bm{z}})$ is the log-density of the noise distribution and $C$ is a normalization constant. For instance, the Gaussian distribution with diagonal covariance matrix and the corresponding Gaussian kernel are obtained with $\nu({\bm{z}})=\frac{1}{2}\|{\bm{z}}\|^{2}_{2}$ and $C=\sqrt{2\pi}^{M}$.