第 1 章 引言Chapter 1 Introduction

在可微编程中，与常规计算机编程一样，程序被定义为基本操作的组合，形成一张计算图。关键区别在于，如图 1.1 所示，程序（例如神经网络）包含可从数据中调整的参数，并能够利用自动微分（autodiff）进行端到端求导。通常假定程序定义了一个数学上有效的函数（又称纯函数）：该函数对相同参数应返回相同的值，且不应有任何副作用。此外，该函数应具有良定义的导数，以确保其可用于基于梯度的优化算法。因此，可微编程不仅是对程序求导的艺术，也是设计有意义的可微程序的艺术。In differentiable programming, as in regular computer programming, a program is defined as the composition of elementary operations, forming a computation graph. The key difference is that, as illustrated in Fig. 1.1, the program (such as a neural network) contains parameters that can be adjusted from data and can be differentiated end-to-end using automatic differentiation (autodiff). Typically, it is assumed that the program defines a mathematically valid function (a.k.a. pure function): the function should return identical values for identical arguments and should not have any side effects. Moreover, the function should have well-defined derivatives, ensuring that it can be used in a gradient-based optimization algorithm. Therefore, differentiable programming is not only the art of differentiating through programs but also of designing meaningful differentiable programs.

机器学习通常归结为优化某个目标函数，该函数是损失函数与模型（网络）函数的组合。无导数优化被称为零阶优化。它只假定我们能够评估希望优化的目标函数。遗憾的是，众所周知它会遭遇维数灾难，即只能扩展到低维问题，例如维度小于 $10$ 的问题。而基于导数的优化则高效得多，能够扩展到数百万或数十亿个参数。使用一阶与二阶导数的算法分别被称为一阶算法与二阶算法。Machine learning typically boils down to optimizing a certain objective function, which is the composition of a loss function and a model (network) function. Derivative-free optimization is called zero-order optimization. It only assumes that we can evaluate the objective function that we wish to optimize. Unfortunately, it is known to suffer from the curse of dimensionality, i.e., it only scales to low-dimensional problems, such as less than $10$ dimensions. Derivative-based optimization, on the other hand, is much more efficient and can scale to millions or billions of parameters. Algorithms that use first and second derivatives are known as first-order and second-order algorithms, respectively.

在自动微分革命之前，研究人员与从业者需要手工实现他们希望优化的函数的梯度。对于复杂函数，手工推导梯度可能变得非常繁琐。此外，每当函数发生变化（例如尝试一个新想法），梯度就需要重新推导。自动微分之所以是颠覆性的，是因为它让用户能够快速而富有创造性地针对其任务试验各种函数。图 1.2 给出了一个 JAX 代码示例 [34]。Before the autodiff revolution, researchers and practitioners needed to manually implement the gradient of the functions they wished to optimize. Manually deriving gradients can become very tedious for complicated functions. Moreover, every time the function is changed (for example, for trying out a new idea), the gradient needs to be re-derived. Autodiff is a game changer because it allows users to focus on quickly and creatively experimenting with functions for their tasks. An example of JAX code [34] is given in Fig. 1.2.

尽管深度学习与可微编程之间显然存在重叠，但二者关注点不同。深度学习研究由多层构成的人工神经网络，能够学习数据的中间表示。人们已提出具有各种归纳偏置的神经网络架构。例如，卷积神经网络是为图像设计的，而 transformer 是为序列设计的。另一方面，可微编程研究的是设计复杂程序并对其求导的技术。它的用处不止于深度学习：例如在强化学习、概率编程以及一般意义上的科学计算中。While there is clearly overlap between deep learning and differentiable programming, their focus is different. Deep learning studies artificial neural networks composed of multiple layers, able to learn intermediate representations of the data. Neural network architectures have been proposed with various inductive biases. For example, convolutional neural networks are designed for images, and transformers are designed for sequences. On the other hand, differentiable programming studies the techniques for designing complex programs and differentiating through them. It is useful beyond deep learning: for instance in reinforcement learning, probabilistic programming and scientific computing in general.

尽管自动微分是可微编程的关键要素，但它并非唯一要素。可微编程还关注有原则的可微操作的设计。事实上，可微编程的许多研究都致力于使经典计算机编程操作与自动微分兼容。正如我们将看到的，许多可微松弛都可以在概率框架中加以解释。本书的一个核心主题是优化、概率与微分之间的相互作用。微分对优化有用，反过来，优化也可用于设计可微算子。While autodiff is a key ingredient of differentiable programming, it is not the only one. Differentiable programming is also concerned with the design of principled differentiable operations. In fact, much research on differentiable programming has been devoted to making classical computer programming operations compatible with autodiff. As we shall see, many differentiable relaxations can be interpreted in a probabilistic framework. A core theme of this book is the interplay between optimization, probability and differentiation. Differentiation is useful for optimization and conversely, optimization can be used to design differentiable operators.

计算机编程提供了控制流、数据结构和标准库等强大工具，使用户能够构建复杂程序来解决错综复杂的问题。我们的长期愿景是实现传统编程与可微编程之间的对等，使程序员能够利用他们所熟悉的全套工具，无缝地表达可微程序（例如神经网络）。然而，如前所述，可微编程并不仅仅是将自动微分应用于现有代码。程序的设计必须考虑到可微性。这通常归结为在程序或其组件上诱导出一个概率分布。尽管要完全实现这一雄心勃勃的目标仍有大量工作要做，但我们希望本书能提供一个坚实的基础。Computer programming offers powerful tools like control flows, data structures, and standard libraries, enabling users to construct complex programs for solving intricate problems. Our long-term vision is to achieve parity between traditional and differentiable programming, empowering programmers to seamlessly express differentiable programs (such as neural networks) using the full suite of tools they are accustomed to. However, as discussed earlier, differentiable programming is not simply a matter of applying automatic differentiation to existing code. Programs must be designed with differentiability in mind. This usually comes down to inducing a probability distribution over the program or its components. While significant work remains to fully realize this ambitious goal, we hope this book offers a solid foundation.

尽管神经网络与自动微分已存在数十年，“可微编程”这一术语却出现得较晚。[148] 讨论了神经网络架构与函数式编程中高阶函数之间的类比，并将其称为“一种新的编程方式”。[49] 撰写了一篇题为“可微编程”的文章，指出存在一种范式：程序通过微分来学习细节，同时具备函数式编程的表达能力。[156] 在 POPL 2018 上作了题为“微分编程语言的一些原则”的主题演讲。“可微编程”与“软件 2.0”这两个术语由 [117] 和 [104] 等人推广开来。从这个视角看，自动微分框架不应仅仅被视为库，而应被视为嵌入在现有编程语言（如 Python）中的领域特定语言（DSL）。另见 [98] 的综述。While neural networks and autodiff have existed for several decades, the term “differentiable programming” is more recent. [148] discussed analogies between neural network architectures and higher-order functions in functional programming, and referred to it as a “new kind of programming”. [49] wrote an essay titled “differentiable programming”, recognizing a paradigm where programs learn details through differentiation, with the expressiveness of functional programming. [156] gave a keynote talk titled “Some principles of differential programming languages” at POPL 2018. The “differentiable programming” and “software 2.0” terms were popularized, among others, by [117] and [104]. From this perspective, autodiff frameworks can be seen, not merely as libraries, but as domain-specific languages (DSLs) embedded into an existing programming language, such as Python. See also [98] for a review.