第 2 章 微分Chapter 2 Differentiation

为了研究函数，我们需要捕捉函数在点附近的无穷小变化，这由极限的概念来定义。To study functions, we need to capture their infinitesimal variations around points as defined by the notion of limit.

我们也可以将 $f(v)\to c$ 写作 $v\to w$。极限在加法与乘法下保持不变。即，若 $\lim_{v\rightarrow w}f(v)=c$ 且 $\lim_{v\rightarrow w}g(v)=d$，则对任意 $a,b\in\mathbb{R}$ 与 $(fg)(w)\coloneqq f(w)g(w)$，记 $(af+bg)(w)\coloneqq af(w)+bg(w)$，根据极限的定义，我们有 $\lim_{v\rightarrow w}(af+bg)(v)=ac+bd$ 与 $\lim_{v\rightarrow w}(fg)(v)=cd$。极限在加法与数乘下保持不变，这通常被称为极限的线性性，后续许多定义都继承了这一性质。We can also write $f(v)\to c$ as $v\to w$. Limits are preserved under additions and multiplications. Namely, if $\lim_{v\rightarrow w}f(v)=c$ and $\lim_{v\rightarrow w}g(v)=d$, then denoting $(af+bg)(w)\coloneqq af(w)+bg(w)$ for any $a,b\in\mathbb{R}$ and $(fg)(w)\coloneqq f(w)g(w)$, we have by definition of the limit, $\lim_{v\rightarrow w}(af+bg)(v)=ac+bd$ and $\lim_{v\rightarrow w}(fg)(v)=cd$. The preservation of the limit under addition and multiplication by a scalar is generally referred to as the linearity of the limit, a property that many definitions in the sequel inherit.

借助极限的概念，我们已经可以刻画一类“性态良好”的函数：在任何点处的极限都等于该点处函数值的函数。满足这一性质的函数称为连续函数。With the notion of limit, we can already delineate a class of “well-behaved” functions: functions whose limit at any point equals the value of the function at that point. Functions satisfying this property are called continuous.

尽管连续性这一概念看似是一个温和的假设，但机器学习中常用的若干函数，例如 Heaviside 阶跃函数（见图 2.2 左图），并不连续，需要特殊处理。Although the notion of continuity appears to be a benign assumption, several functions commonly used in machine learning, such as the Heaviside step function (displayed in the left panel of Fig. 2.2), are not continuous and require special treatment.

现在我们解释导数。考虑一个函数 $f:\mathbb{R}\rightarrow\mathbb{R}$。如图 2.1 所示，该函数在区间 $[w_{0},w_{0}+\delta]$ 上的值可以用其取值 $f(w_{0})$ 与 $f(w_{0}+\delta)$ 之间的割线来近似，这是一条斜率为 $(f(w_{0}+\delta)-f(w_{0}))/\delta$ 的线性函数。在 $w_{0}$ 附近的无穷小变化 $\delta$ 的极限下，割线收敛到 $f$ 在 $w_{0}$ 处的切线，由此得到的斜率定义了 $f$ 在 $w_{0}$ 处的导数。下面的定义将这一直观形式化。We now explain derivatives. Consider a function $f:\mathbb{R}\rightarrow\mathbb{R}$. As illustrated in Fig. 2.1, its value on an interval $[w_{0},w_{0}+\delta]$ can be approximated by the secant between its values $f(w_{0})$ and $f(w_{0}+\delta)$, a linear function with slope $(f(w_{0}+\delta)-f(w_{0}))/\delta$. In the limit of an infinitesimal variation $\delta$ around $w_{0}$, the secant converges to the tangent of $f$ at $w_{0}$ and the resulting slope defines the derivative of $f$ at $w_{0}$. The definition below formalizes this intuition.

若 $f$ 在给定的 $w$ 处可微，则它在该点 $w$ 处必然连续，如下面的命题所示。连续函数在给定点 $w$ 处不可微，通常表现为一个折角，如图 2.2 所示。If $f$ is differentiable at a given $w$, then it is necessarily continuous at $w$ as shown in the following proposition. Non-differentiability of a continuous function at a given point $w$ is generally illustrated by a kink, as shown in Fig. 2.2.

导数 $f^{\prime}$ 是 $f$ 在 $w$ 处切线的斜率，因而它除了能够构造 $f$ 在 $w$ 邻域内的线性近似之外，还能揭示 $f$ 在 $w$ 附近的单调性。若 $f^{\prime}(w)$ 为正，则函数在 $w$ 附近递增；反之，若 $f^{\prime}(w)$ 为负，则函数递减。这类信息可用于设计梯度下降等迭代算法：该类算法通过计算形如 $w_{t+1}=w_{t}-\gamma f^{\prime}(w_{t})$（其中 $\gamma>0$）的迭代值，沿着 $f$ 在 $w_{t}$ 附近的下降方向不断移动，以最小化 $f$。In addition to enabling the construction of a linear approximation of $f$ in a neighborhood of $w$, since it is the slope of the tangent of $f$ at $w$, the derivative $f^{\prime}$ informs us about the monotonicity of $f$ around $w$. If $f^{\prime}(w)$ is positive, the function is increasing around $w$. Conversely, if $f^{\prime}(w)$ is negative, the function is decreasing. Such information can be used to develop iterative algorithms such as gradient descent, which seeks to minimize $f$ by computing iterates of the form $w_{t+1}=w_{t}-\gamma f^{\prime}(w_{t})$ for $\gamma>0$, thereby moving along descent directions of $f$ around $w_{t}$.

对于若干初等函数，例如 $w^{n}$、$e^{w}$、$\ln w$、$\cos w$ 或 $\sin w$，其导数均可直接通过应用式 2.1 中导数的定义求得，下面我们对此加以说明。For several elementary functions such as $w^{n}$, $e^{w}$, $\ln w$, $\cos w$ or $\sin w$, their derivatives can be obtained directly by applying the definition of the derivative in Eq. 2.1 as we now illustrate.

给定 $w\in\mathbb{R}$ 以及两个函数 $f:\mathbb{R}\rightarrow\mathbb{R}$ 与 $g:\mathbb{R}\rightarrow\mathbb{R}$，只要 $f$ 与 $g$ 在 $w$ 处的可微性满足适当条件，便可根据导数的定义容易地推出对 $f$ 与 $g$ 所做初等运算（如求和、求积或复合）的导数。例如，若 $f$ 与 $g$ 的导数在 $w$ 处存在，则其加权和或乘积的导数存在，并满足以下法则For a given $w\in\mathbb{R}$ and two functions $f:\mathbb{R}\rightarrow\mathbb{R}$ and $g:\mathbb{R}\rightarrow\mathbb{R}$, the derivative of elementary operations on $f$ and $g$ such as their sums, products or compositions can easily be derived from the definition of the derivative, under appropriate conditions on the differentiability of $f$ and $g$ at $w$. For example, if the derivatives of $f$ and $g$ exist at $w$, then the derivatives of their weighted sum or product exist, and satisfy the rules

$$\displaystyle\forall a,b\in\mathbb{R},\ (af+bg)^{\prime}(w)$$	$$\displaystyle=af^{\prime}(w)+bg^{\prime}(w)$$	（线性性）(Linearity)
$$\displaystyle(fg)^{\prime}(w)$$	$$\displaystyle=f^{\prime}(w)g(w)+f(w)g^{\prime}(w),$$	（乘积法则）(Product rule)

其中 $(fg)(w)\coloneqq f(w)g(w)$。线性性可直接由极限的线性性验证。至于乘积法则，采用小 $o$ 记号，当 $\delta\rightarrow 0$ 时，我们有where $(fg)(w)\coloneqq f(w)g(w)$. The linearity can be verified directly from the linearity of the limits. For the product rule, in little $o$ notation, we have, as $\delta\rightarrow 0$,

$$\displaystyle(fg)(w+\delta)$$	$$\displaystyle=(f(w)+f^{\prime}(w)\delta+o(\delta))(g(w)+g^{\prime}(w)\delta+o(\delta))$$
	$$\displaystyle=f(w)g(w)+f^{\prime}(w)g(w)\delta+f(w)g^{\prime}(w)\delta+o(\delta),$$

由此得证。hence the result.

若 $g$ 在 $w$ 处的导数与 $f$ 在 $g(w)$ 处的导数均存在，则复合函数 $(f\circ g)(w)\coloneqq f(g(w))$ 在 $w$ 处的导数存在，且由下式给出If the derivatives of $g$ at $w$ and of $f$ at $g(w)$ exist, then the derivative of the composition $(f\circ g)(w)\coloneqq f(g(w))$ at $w$ exists and is given by

$$\displaystyle\hskip 54.0pt(f\circ g)^{\prime}(w)=f^{\prime}(g(w))g^{\prime}(w).$$

（链式法则）(Chain rule)

我们将在命题 2.2.11 中对这一结果给出更一般的证明。正如后文所示，线性性与乘积法则可视为链式法则的推论，因此链式法则构成了微分学的基石。We prove this result more generally in Proposition 2.2.11. As seen in the sequel, the linearity and the product rule can be seen as byproducts of the chain rule, making the chain rule the cornerstone of differentiation.

考虑一个可由初等函数的和、积或复合表示的函数，例如 $f(w)\coloneqq e^{w}\ln w+\cos w^{2}$。将 $f$ 分解为初等运算与初等函数，再应用上述法则，即可计算其导数。Consider a function that can be expressed using sums, products or compositions of elementary functions, such as $f(w)\coloneqq e^{w}\ln w+\cos w^{2}$. Its derivative can be computed by applying the aforementioned rules on the decomposition of $f$ into elementary operations and functions.

这一过程完全是机械性的，因而适合交由自动化流程完成，这正是第 8 章所介绍的自动微分的主要思想。Such a process is purely mechanical and lends itself to an automated procedure, which is the main idea of automatic differentiation presented in Chapter 8.

导数的概念最早由牛顿与莱布尼茨于 18 世纪各自独立提出 [9]。后者将导数视为无穷小变化量之商。也就是说，记 $u=f(w)$ 为通过 $f$ 依赖于 $w$ 的变量，莱布尼茨便将 $f$ 的导数视为如下商The notion of derivative was first introduced independently by Newton and Leibniz in the 18^th century [9]. The latter considered derivatives as the quotient of infinitesimal variations. Namely, denoting $u=f(w)$ a variable depending on $w$ through $f$, Leibniz considered the derivative of $f$ as the quotient

$$f^{\prime}=\frac{du}{dw}\quad\text{with}\quad f^{\prime}(w)=\left.\frac{du}{dw}\right|_{w}$$

其中 $du$ 与 $dw$ 分别表示 $u$ 与 $w$ 的无穷小变化量，符号 $|_{w}$ 则表示在给定点 $w$ 处对导数求值。这一记号简化了莱布尼茨最早发现的链式法则 [170] 的表述，因为对于 $v=g(w)$ 与 $u=f(v)$，我们有where $du$ and $dw$ denote infinitesimal variations of $u$ and $w$ respectively and the symbol $|_{w}$ denotes the evaluation of the derivative at a given point $w$. This notation simplifies the statement of the chain rule first discovered by Leibniz [170] as we have for $v=g(w)$ and $u=f(v)$

$$\frac{du}{dw}=\frac{du}{dv}\cdot\frac{dv}{dw}.$$

这暗示在考虑复合时导数会相乘。求值时，莱布尼茨记号下的链式法则可还原出上文给出的公式：This hints that derivatives are multiplied when considering compositions. At evaluation, the chain rule in Leibniz notation recovers the formula presented above as

$$\left.\frac{du}{dw}\right|_{w}=\left.\frac{du}{dv}\right|_{g(w)}\left.\frac{dv}{dw}\right|_{w}=f^{\prime}(g(w))g^{\prime}(w)=(f\circ g)^{\prime}(w).$$

莱布尼茨记号能将链式法则表达为商之乘积，这一能力使其在数个世纪以来广受欢迎，尤其是在力学领域 [9]。莱布尼茨记号背后的原理，即“无穷小变化量”这一概念，因其潜在的逻辑问题而受到后来数学家的质疑 [9]。由欧拉首次引入、后经拉格朗日进一步推广的记号 $f^{\prime}(w)$ [40] 随之在众多数学教材中占据主流。无穷小变化量的概念已借助超实数集得到严格定义。超实数集将每个数视为非无穷小部分与无穷小部分之和，从而扩展了实数集 [94]。无穷小变化量的形式化还通过对偶数这一概念，进一步支撑了自动微分算法的发展。The ability of Leibniz’s notation to capture the chain rule as a mere product of quotients made it popular throughout the centuries, especially in mechanics [9]. The rationale behind Leibniz’s notation, the concept of “infinitesimal variations”, was questioned by later mathematicians for its potential logical issues [9]. The notation $f^{\prime}(w)$ first introduced by Euler and further popularized by Lagrange [40] has then taken over in numerous mathematical textbooks. The concept of infinitesimal variations has been rigorously defined by using the set of hyperreal numbers. They extend the set of real numbers by considering each number as a sum of a non-infinitesimal part and an infinitesimal part [94]. The formalism of infinitesimal variations further underlies the development of automatic differentiation algorithms through the concept of dual numbers.

现在我们考虑一个输入为多维的 $f:\mathbb{R}^{P}\rightarrow\mathbb{R}$ 函数，其输入为 ${\bm{w}}\coloneqq(w_{1},\ldots,w_{P})\in\mathbb{R}^{P}$。机器学习中最重要的例子，是这样一个函数：它将神经网络的参数 ${\bm{w}}\in\mathbb{R}^{P}$ 映射为 $\mathbb{R}$ 中的损失值。$f$ 的变化需要沿特定方向定义，例如 $f$ 在 ${\bm{w}}\in\mathbb{R}^{P}$ 附近沿方向 ${\bm{v}}\in\mathbb{R}^{P}$ 发生大小为 $\delta>0$ 的变化 $f({\bm{w}}+\delta{\bm{v}}){-}f({\bm{w}})$。这一考虑自然引出了方向导数的定义。Let us now consider a function $f:\mathbb{R}^{P}\rightarrow\mathbb{R}$ with multi-dimensional input ${\bm{w}}\coloneqq(w_{1},\ldots,w_{P})\in\mathbb{R}^{P}$. The most important example in machine learning is a function which, to the parameters ${\bm{w}}\in\mathbb{R}^{P}$ of a neural network, associates a loss value in $\mathbb{R}$. Variations of $f$ need to be defined along specific directions, such as the variation $f({\bm{w}}+\delta{\bm{v}}){-}f({\bm{w}})$ of $f$ around ${\bm{w}}\in\mathbb{R}^{P}$ in the direction ${\bm{v}}\in\mathbb{R}^{P}$ by an amount $\delta>0$. This consideration naturally leads to the definition of the directional derivative.

我们采用记号 $[{\bm{v}}]$，以强调对给定的输入 ${\bm{w}}$，可将 ${\bm{v}}\mapsto\partial f({\bm{w}})[{\bm{v}}]$ 视为一个函数。这对于定义 $f$ 在 ${\bm{w}}$ 处的可微性（定义 2.2.3），以及后文定义线性映射（2.3 节），都是必不可少的。We use the notation $[{\bm{v}}]$ to emphasize that, for a given input ${\bm{w}}$, we can see ${\bm{v}}\mapsto\partial f({\bm{w}})[{\bm{v}}]$ as a function. This is essential to define the differentiability of $f$ (Definition 2.2.3) at ${\bm{w}}$ and to later define linear maps (Section 2.3).

方向导数的一个例子是：求函数 $f$ 在 ${\bm{w}}$ 处沿任一标准方向的导数One example of directional derivative consists in computing the derivative of a function $f$ at ${\bm{w}}$ in any of the canonical directions

$${\bm{e}}_{i}\coloneqq(0,\ldots,0,\underbrace{1}_{i},0,\ldots,0).$$

据此便可定义偏导数的概念，对 $i\in[P]$ 记作This allows us to define the notion of partial derivatives, denoted for $i\in[P]$

$$\partial_{i}f({\bm{w}})\coloneqq\partial f({\bm{w}})[{\bm{e}}_{i}]=\lim_{\delta\rightarrow 0}\frac{f({\bm{w}}+\delta{\bm{e}}_{i})-f({\bm{w}})}{\delta}.$$

用莱布尼茨记号，这也可写作 $\partial_{i}f({\bm{w}})=\frac{\partial f({\bm{w}})}{\partial w_{i}}$ 或 $\partial_{i}f({\bm{w}})=\partial_{w_{i}}f({\bm{w}})$。由于只沿函数的第 $i$ 个坐标移动，偏导数相当于在其余坐标均固定于 $w_{i}$ 的情形下，对 $\omega_{i}$ 附近的函数 $\omega_{i}\mapsto f(w_{1},\ldots,\omega_{i},\ldots,w_{P})$ 求导。This is also denoted in Leibniz’s notation as $\partial_{i}f({\bm{w}})=\frac{\partial f({\bm{w}})}{\partial w_{i}}$ or $\partial_{i}f({\bm{w}})=\partial_{w_{i}}f({\bm{w}})$. By moving along only the $i$^th coordinate of the function, the partial derivative is akin to differentiating the function $\omega_{i}\mapsto f(w_{1},\ldots,\omega_{i},\ldots,w_{P})$ around $\omega_{i}$, letting all other coordinates fixed at their values $w_{i}$.

下面引入梯度向量，它汇集了各个偏导数。在此之前，先回顾线性映射与线性泛函的定义。We now introduce the gradient vector, which gathers the partial derivatives. We first recall the definitions of linear map and linear form.

线性性在函数的可微性中起着关键作用。Linearity plays a crucial role in the differentiability of a function.

现在可以引入梯度了。We can now introduce the gradient.

此处 $\langle\cdot,\cdot\rangle$ 表示内积，其定义见 2.3.2 节关于欧几里得空间的讨论。Here, $\langle\cdot,\cdot\rangle$ denotes the inner product. We provide its definition in Euclidean spaces in Section 2.3.2.

在上述定义中，梯度之所以能用来计算方向导数，不过是 $\partial f({\bm{w}})[{\bm{v}}]$ 关于 ${\bm{v}}$ 的线性性的一个推论。然而在后文所介绍的更为抽象的情形中，梯度正是通过这一性质直接定义的。In the definition above, the fact that the gradient can be used to compute the directional derivative is a mere consequence of the linearity of $\partial f({\bm{w}})[{\bm{v}}]$ w.r.t. ${\bm{v}}$. However, in more abstract cases presented in later sections, the gradient is defined directly through this property.

作为一个简单的例子，任何形如 $f({\bm{w}})=\langle{\bm{a}},{\bm{w}}\rangle=\sum_{i=1}^{P}a_{i}w_{i}$ 的线性函数都是可微的，因为对任意 ${\bm{v}}$ 都有 $(\langle{\bm{a}},{\bm{w}}+{\bm{v}}\rangle-\langle{\bm{a}},{\bm{w}}\rangle-\langle{\bm{a}},{\bm{v}}\rangle)/\|{\bm{v}}\|_{2}=0$，特别是当 $\|{\bm{v}}\|\rightarrow 0$ 时也成立。此外，其梯度自然由 $\nabla f({\bm{w}})={\bm{a}}$ 给出。As a simple example, any linear function of the form $f({\bm{w}})=\langle{\bm{a}},{\bm{w}}\rangle=\sum_{i=1}^{P}a_{i}w_{i}$ is differentiable as we have $(\langle{\bm{a}},{\bm{w}}+{\bm{v}}\rangle-\langle{\bm{a}},{\bm{w}}\rangle-\langle{\bm{a}},{\bm{v}}\rangle)/\|{\bm{v}}\|_{2}=0$ for any ${\bm{v}}$ and in particular for $\|{\bm{v}}\|\rightarrow 0$. Moreover, its gradient is naturally given by $\nabla f({\bm{w}})={\bm{a}}$.

更一般地，若要证明一个函数可微并求出其梯度，一种方法是在 ${\bm{v}}={\mathbf{0}}$ 附近对 $f({\bm{w}}+{\bm{v}})$ 作近似。如果我们能找到一个向量 ${\bm{g}}$，使得More generally, to show that a function is differentiable and find its gradient, one approach is to approximate $f({\bm{w}}+{\bm{v}})$ around ${\bm{v}}={\mathbf{0}}$. If we can find a vector ${\bm{g}}$ such that

$$f({\bm{w}}+{\bm{v}})=f({\bm{w}})+\langle{\bm{g}},{\bm{v}}\rangle+o(\|{\bm{v}}\|_{2}),$$

那么 $f$ 在 ${\bm{w}}$ 处可微，因为 $\langle{\bm{g}},\cdot\rangle$ 是线性的。此外，此时 ${\bm{g}}$ 正是 $f$ 在 ${\bm{w}}$ 处的梯度。then $f$ is differentiable at ${\bm{w}}$, since $\langle{\bm{g}},\cdot\rangle$ is linear. Moreover, ${\bm{g}}$ is then the gradient of $f$ at ${\bm{w}}$.

下一个例子说明如何计算 logistic 损失的梯度，并验证其可微性。The next example illustrates how to compute the gradient of the logistic loss and validates its differentiability.