Algorithmic differentiation for domain specific languages in C++ with expression templates

TL;DR

This work proposes a new AD tool that is tailored to such situations and demonstrates that this approach provides performance comparable to that of other specializations.

Abstract

The application of operator overloading algorithmic differentiation (AD) to computer programs in order to compute the derivative is quite common. But, the replacement of the underlying computational floating point type with the specialized type of an AD tool has two problems. First, the memory structure of the program is changed and floating-point data is interleaved with identifiers from AD. This prevents the compiler from performing optimizations such as SIMD optimizations. Second, the AD tool does not see any domain-specific operations, e.,g. linear algebra operations, that the program uses. This prevents the AD tool from using specialized algorithms in such places. We propose a new AD tool that is tailored to such situations. The memory structure of the primal data is retained by associating an identifier with each entity, e.,g. matrix, and not with each floating point value, e.,g. element of the matrix. Operations on such entities can then be annotated and a generator is used to create the AD overloads. We demonstrate that this approach provides performance comparable to that of other specializations. In addition, the run-time factor is below the theoretical 4.5 of reverse AD for programs that are written purely with linear algebra entities and operations.

Figures (14)

• Figure 1: Graph representation of the generated expression template for $w = (a + b) * (c - d)$
• Figure 2: Graph representation of the generated expression template for Equation \ref{['eq:example_expression']}.
• Figure 3: Memory layout of the tape. Global vectors are allocated for each registered DSL entity. Sequential access data is kept general such that arbitrary data can be stored.
• Figure 4: Definition of the overloaded AD type.
• Figure 5: Reconstructed expression graph for the example Equation \ref{['eq:example_expression']}. A wrapper node for the old value is inserted and the leafs are exchanged with id nodes.