PRDP: Progressively Refined Differentiable Physics

TL;DR

This work tackles the high computational cost of differentiating through iterative physics solvers in neural network training. It introduces Progressively Refined Differentiable Physics (PRDP), which starts training with coarse physics and adaptively refines the solver during training, stopping once accuracy plateaus to maintain network performance with far fewer solver iterations. PRDP combines progressive refinement (PR) and incomplete convergence (IC) savings and supports both implicit and unrolled differentiation, demonstrating substantial reductions in inner iterations and training time across Poisson inverse problems, linear/nonlinear heat emulators, and Navier–Stokes neural-hybrid emulators. Across 1D–3D problems, PRDP achieves up to 86% fewer inner iterations and up to 78% reductions in wall-clock training time, while preserving or improving validation accuracy. The method promises broad impact for training models that couple neural networks with differentiable solvers in physics-informed learning and scientific computing.

Abstract

The physics solvers employed for neural network training are primarily iterative, and hence, differentiating through them introduces a severe computational burden as iterations grow large. Inspired by works in bilevel optimization, we show that full accuracy of the network is achievable through physics significantly coarser than fully converged solvers. We propose Progressively Refined Differentiable Physics (PRDP), an approach that identifies the level of physics refinement sufficient for full training accuracy. By beginning with coarse physics, adaptively refining it during training, and stopping refinement at the level adequate for training, it enables significant compute savings without sacrificing network accuracy. Our focus is on differentiating iterative linear solvers for sparsely discretized differential operators, which are fundamental to scientific computing. PRDP is applicable to both unrolled and implicit differentiation. We validate its performance on a variety of learning scenarios involving differentiable physics solvers such as inverse problems, autoregressive neural emulators, and correction-based neural-hybrid solvers. In the challenging example of emulating the Navier-Stokes equations, we reduce training time by 62%.

Figures (28)

• Figure 1: A neural network training pipeline using a differentiable physics solver $\mathcal{P}_K$. Black and red arrows show the forward and backward passes, respectively. As solver iterations $K$ grows, the cost of passes through $\mathcal{P}_K$ becomes severe.
• Figure 2: Progressively Refined Differentiable Physics (PRDP) reduces the training time of neural networks containing numerical solver components (c). The fidelity of iterative components is increased only if validation metrics plateau. This leads to savings by using fewer iterations in the beginning (PR savings in (b)) and by ending at a refinement level significantly below full fidelity (IC savings in (b)). The achieved validation error is identical (a).
• Figure 3: Progressively refining the differentiable physics during outer optimization of a Poisson inverse problem achieves full convergence of the parameter with fewer cumulative iterations of the physics solver, leading to progressive refinement (PR) savings.
• Figure 4: Network accuracy does not improve beyond a refinement level of differentiable physics ($K_\text{max}$) significantly lower than full convergence ($K_\epsilon$) constituting incomplete convergence (IC) savings.
• Figure 5: Top: the typical training progress of a neural network supported by PRDP, showing the ratios $r$ and $r_c$. Bottom: a simplified flowchart representation of the PRDP control algorithm.