The Elements of
Differentiable Programming

摘要Abstract

人工智能近来取得了显著进展，这得益于大模型、海量数据集、加速硬件，以及——同样不可忽视的——可微编程的变革性力量。这一新的编程范式使得复杂计算机程序（包括含有控制流与数据结构的程序）能够端到端求导，从而使基于梯度的程序参数优化成为可能。Artificial intelligence has recently experienced remarkable advances, fueled by large models, vast datasets, accelerated hardware, and, last but not least, the transformative power of differentiable programming. This new programming paradigm enables end-to-end differentiation of complex computer programs (including those with control flows and data structures), making gradient-based optimization of program parameters possible.

作为一种新兴范式，可微编程建立在计算机科学与应用数学的若干领域之上，包括自动微分、图模型、优化与统计学。本书对可微编程所需的基础概念进行了全面梳理。我们采用两个主要视角，即优化视角与概率视角，并在二者之间建立清晰的类比。As an emerging paradigm, differentiable programming builds upon several areas of computer science and applied mathematics, including automatic differentiation, graphical models, optimization and statistics. This book presents a comprehensive review of the fundamental concepts useful for differentiable programming. We adopt two main perspectives, that of optimization and that of probability, with clear analogies between the two.

可微编程不仅仅是对程序求导，更是对旨在可微的程序进行深思熟虑的设计。通过使程序可微，我们必然引入程序执行上的概率分布，从而提供了一种量化程序输出不确定性的手段。Differentiable programming is not merely the differentiation of programs, but also the thoughtful design of programs intended for differentiation. By making programs differentiable, we inherently introduce probability distributions over their execution, providing a means to quantify the uncertainty associated with program outputs.