Implicit differentiation with second-order derivatives and benchmarks in finite-element-based differentiable physics

TL;DR

This work develops a practical framework for second-order implicit differentiation in FEM-based differentiable physics, deriving and implementing Hessian-vector products via Jacobian-vector and vector-Jacobian products. By operating under the discretize-then-optimize paradigm, the authors bridge adjoint-based Hessian methods with modern automatic differentiation and validate accuracy against finite differences through Taylor remainder tests. Four benchmarks spanning linear/nonlinear, 2D/3D, and single/coupled variables demonstrate that Newton-CG with exact Hessians accelerates convergence for nonlinear inverse problems, while L-BFGS-B remains effective for linear tasks. The approach enables more robust and faster PDE-constrained optimization in differentiable physics engines, with open-source tooling to facilitate adoption and future sparsity-focused acceleration.

Abstract

Differentiable programming is revolutionizing computational science by enabling automatic differentiation (AD) of numerical simulations. While first-order gradients are well-established, second-order derivatives (Hessians) for implicit functions in finite-element-based differentiable physics remain underexplored. This work bridges this gap by deriving and implementing a framework for implicit Hessian computation in PDE-constrained optimization problems. We leverage primitive AD tools (Jacobian-vector product/vector-Jacobian product) to build an algorithm for Hessian-vector products and validate the accuracy against finite difference approximations. Four benchmarks spanning linear/nonlinear, 2D/3D, and single/coupled-variable problems demonstrate the utility of second-order information. Results show that the Newton-CG method with exact Hessians accelerates convergence for nonlinear inverse problems (e.g., traction force identification, shape optimization), while the L-BFGS-B method suffices for linear cases. Our work provides a robust foundation for integrating second-order implicit differentiation into differentiable physics engines, enabling faster and more reliable optimization.

Figures (17)

• Figure 1: Differentiable programming breaks the boundary between deep learning and differentiable physics.
• Figure 2: The forward prediction part of the model problem for implicit differentiation.
• Figure 3: Relative difference $e_{{\boldsymbol v}}$ of Hessian-vector product evaluations between the proposed implicit differentiation approach and the finite difference approach. In each subfigure, 100 samples are used to generate the histogram plot.
• Figure 4: Relative difference $e_{s}$ of Hessian-vector product evaluations between the proposed implicit differentiation approach and the finite difference approach. In each subfigure, 100 samples are used to generate the histogram plot.
• Figure 5: Taylor remainder test. As expected, the zeroth-order expansion of the residual achieves a first order convergence, the first-order expansion achieves a second order convergence, and the second-order expansion achieves a third order convergence.