Automatic differentiation in machine learning: a survey

TL;DR

Automatic differentiation provides exact derivatives by propagating information through program execution, unifying forward and reverse accumulation modes and underpinning backpropagation as a special case. The survey clarifies differences from numerical and symbolic differentiation, reviews implementation strategies (elemental libraries, source-code transformation, and operator overloading), and covers AD applications across gradient-based optimization, deep learning, vision, NLP, and probabilistic inference. It highlights practical considerations (memory, numerical stability, perturbation issues) and outlines future directions, including nested AD and differentiable programming, to advance ML tooling and architectures. Overall, the work positions general-purpose AD as a foundational tool for scalable, flexible differentiable computation in ML.

Abstract

Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in machine learning. Automatic differentiation (AD), also called algorithmic differentiation or simply "autodiff", is a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs. AD is a small but established field with applications in areas including computational fluid dynamics, atmospheric sciences, and engineering design optimization. Until very recently, the fields of machine learning and AD have largely been unaware of each other and, in some cases, have independently discovered each other's results. Despite its relevance, general-purpose AD has been missing from the machine learning toolbox, a situation slowly changing with its ongoing adoption under the names "dynamic computational graphs" and "differentiable programming". We survey the intersection of AD and machine learning, cover applications where AD has direct relevance, and address the main implementation techniques. By precisely defining the main differentiation techniques and their interrelationships, we aim to bring clarity to the usage of the terms "autodiff", "automatic differentiation", and "symbolic differentiation" as these are encountered more and more in machine learning settings.

Figures (6)

• Figure 1: Overview of backpropagation. (a) Training inputs $x_i$ are fed forward, generating corresponding activations $y_i$. An error $E$ between the actual output $y_3$ and the target output $t$ is computed. (b) The error adjoint is propagated backward, giving the gradient with respect to the weights $\nabla_{w_i}E = \left(\frac{\partial E}{\partial w_1},\dots,\frac{\partial E}{\partial w_6}\right)$, which is subsequently used in a gradient-descent procedure. The gradient with respect to inputs $\nabla_{x_i}E$ can be also computed in the same backward pass.
• Figure 2:
• Figure 3: Error in the forward (Eq. \ref{['EquationForwardDifference']}) and center difference (Eq. \ref{['EquationCenterDifference']}) approximations as a function of step size $h$, for the derivative of the truncated logistic map ${f(x)=64x(1 - x)(1 - 2 x)^2 (1 - 8 x + 8 x^2)^2}$. Plotted errors are computed using ${E_{\mathrm{forward}}(h,x_0)=\left|\frac{f(x_0+h)-f(x_0)}{h} - \frac{d}{dx}f(x)|_{x_0}\right|}$ and ${E_{\mathrm{center}}(h,x_0)=\left|\frac{f(x_0+h)-f(x_0-h)}{2h} - \frac{d}{dx}f(x)|_{x_0}\right|}$ at $x_0=0.2$ .
• Figure 4:
• Figure 5: Evaluation time of the Helmholtz free energy function of a mixed fluid, based on the Peng-Robinson equation of state Peng1976, ${f(\mathbf{x}) = R \, T \sum_{i = 0}^{n} \log \frac{x_i}{1 - \mathbf{b^T} \mathbf{x}} - \frac{\mathbf{x^T} \mathbf{A} \mathbf{x}}{\sqrt{8} \mathbf{b^T} \mathbf{x}} \log \frac{1 + (1 + \sqrt{2}) \mathbf{b^T} \mathbf{x}}{1 + (1 - \sqrt{2}) \mathbf{b^T} \mathbf{x}}}$, where $R$ is the universal gas constant, $T$ is the absolute temperature, $\mathbf{b} \in \mathbb{R}^n$ is a vector of constants, $\mathbf{A} \in \mathbb{R}^{n \times n}$ is a symmetric matrix of constants, and $\mathbf{x} \in \mathbb{R}^n$ is the vector of independent variables describing the system. The plots show the evaluation time of $f$ and the gradient $\nabla f$ with numerical differentiation (central difference), forward mode AD, and reverse mode AD, as a function of the number of variables $n$. Reported times are relative to the evaluation time of $f$ with $n=1$. The lower plot uses logarithmic scale for illustrating the behavior for small $n$. Numerical results are given in Table \ref{['TableHelmholtz']}. (Code: http://DiffSharp.github.io/DiffSharp/misc/Benchmarks-h-grad-v0.5.7.fsx)