Optimization Landscapes Learned: Proxy Networks Boost Convergence in Physics-based Inverse Problems

TL;DR

Inverse problems governed by PDEs produce nonconvex, chaotic loss landscapes $\mathcal{L}(Y^*, X_s)$ with local minima and vanishing gradients. We introduce ProxyNNs that model $\mathcal{L}$ from spatio-temporal trajectories and apply regularization to control the landscape's geometric complexity, enabling smoother optimization. The optimization uses a two-step procedure: first minimize on the ProxyNN-predicted landscape using a momentum-based method such as BFGS, then refine on the true loss $\mathcal{L}$ starting from that solution. Across Burgers' equation, Kuramoto–Sivashinsky, and billiards-inspired dynamics, the approach yields improved convergence and lower resimulation error compared to conventional optimization.

Abstract

Solving inverse problems in physics is central to understanding complex systems and advancing technologies in various fields. Iterative optimization algorithms, commonly used to solve these problems, often encounter local minima, chaos, or regions with zero gradients. This is due to their overreliance on local information and highly chaotic inverse loss landscapes governed by underlying partial differential equations (PDEs). In this work, we show that deep neural networks successfully replicate such complex loss landscapes through spatio-temporal trajectory inputs. They also offer the potential to control the underlying complexity of these chaotic loss landscapes during training through various regularization methods. We show that optimizing on network-smoothened loss landscapes leads to improved convergence in predicting optimum inverse parameters over conventional momentum-based optimizers such as BFGS on multiple challenging problems.

Figures (8)

• Figure 1: Schematic representation of formulating the configuration loss landscape $\mathcal{L} = ||Y_s - Y^*||^2$. The Gramacy and Lee function represent a test case for configuration loss landscapes in this figure.
• Figure 2: System trajectories and configuration loss landscapes ($\mathcal{L}$) for non-linear inverse problems in (a,e) Burgers equation, (b,f) for the Kuramoto-Sivashinsky equation, and (c,d) and (g,h) for the Billiards-2D setup respectively. These landscapes (f-h) encounter convergence issues in iterative optimizers due to local minima and regions of sharp and vanishing gradients.
• Figure 3: ProxyNNs predict configuration loss landscapes $\mathcal{L}$.
• Figure 4: Predicted Loss Landscapes (top) and Optimization performance (bottom) of ProxyNNs in the Billiards setup trained with various regularization hyperparameter $\sigma=\{0, 0.050\}$.
• Figure 5: For 256 unique inverse problems for each setup, we report convergence accuracy with varying error thresholds: Regularized ProxyNN results in improved accuracy across the board.