Forward-Mode Automatic Differentiation in Julia

TL;DR

The article introduces ForwardDiff, a Julia package for forward-mode automatic differentiation that leverages JIT compilation to optimize AD-unaware code, enabling higher-order derivatives and support for custom number types. It presents a chunk-mode approach with stack-allocated epsilon vectors to reduce heap allocations and improve memory usage, while allowing nesting of dual numbers to compute higher-order derivatives. Performance results indicate ForwardDiff can outperform reverse-mode autograd on large-dimensional problems and benefits further from multithreading, with competitive performance against C++ implementations. The paper also demonstrates practical integration with JuMP, enabling differentiation of user-defined functions within optimization models, and outlines future improvements in SIMD, perturbation handling, and linear algebra support. Overall, ForwardDiff advances AD in Julia by combining memory-efficient forward-mode, higher-order capabilities, and seamless modeling integration.

Abstract

We present ForwardDiff, a Julia package for forward-mode automatic differentiation (AD) featuring performance competitive with low-level languages like C++. Unlike recently developed AD tools in other popular high-level languages such as Python and MATLAB, ForwardDiff takes advantage of just-in-time (JIT) compilation to transparently recompile AD-unaware user code, enabling efficient support for higher-order differentiation and differentiation using custom number types (including complex numbers). For gradient and Jacobian calculations, ForwardDiff provides a variant of vector-forward mode that avoids expensive heap allocation and makes better use of memory bandwidth than traditional vector mode. In our numerical experiments, we demonstrate that for nontrivially large dimensions, ForwardDiff's gradient computations can be faster than a reverse-mode implementation from the Python-based autograd package. We also illustrate how ForwardDiff is used effectively within JuMP, a modeling language for optimization. According to our usage statistics, 41 unique repositories on GitHub depend on ForwardDiff, with users from diverse fields such as astronomy, optimization, finite element analysis, and statistics. This document is an extended abstract that has been accepted for presentation at the AD2016 7th International Conference on Algorithmic Differentiation.