Learning Gradient Flow: Using Equation Discovery to Accelerate Engineering Optimization

TL;DR

The Learned Gradient Flow (LGF) optimizer, which is equipped to build surrogate models of variable polynomial order in full- or reduced-dimensional spaces at user-defined intervals in the optimization process, is introduced.

Abstract

In this work, we investigate the use of data-driven equation discovery for dynamical systems to model and forecast continuous-time dynamics of unconstrained optimization problems. To avoid expensive evaluations of the objective function and its gradient, we leverage trajectory data on the optimization variables to learn the continuous-time dynamics associated with gradient descent, Newton's method, and ADAM optimization. The discovered gradient flows are then solved as a surrogate for the original optimization problem. To this end, we introduce the Learned Gradient Flow (LGF) optimizer, which is equipped to build surrogate models of variable polynomial order in full- or reduced-dimensional spaces at user-defined intervals in the optimization process. We demonstrate the efficacy of this approach on several standard problems from engineering mechanics and scientific machine learning, including two inverse problems, structural topology optimization, and two forward solves with different discretizations. Our results suggest that the learned gradient flows can significantly expedite convergence by capturing critical features of the optimization trajectory while avoiding expensive evaluations of the objective and its gradient.

Figures (13)

• Figure 1: Scheduled retraining of gradient flow. We alternate between collecting histories for $\mathcal{K}$ iterations from the true optimizer with expensive gradient computations (shaded red) and applying the learned gradient flow, ending at iteration $\mathcal{M}$ (shaded green). Transitioning between these two steps, we must build $\hat{\mathbf{f}}$ as specified in Sec. \ref{['sec:learned_gradient_flow']}. The lines in the plot indicate trajectories of individual optimization variables.
• Figure 2: Convergence of the two optimizers to the minimum of the data loss defining the inverse problem. The optimizer based on the SINDy surrogate for the dynamics tracks closely with standard gradient descent and accelerates the optimization by $200 \%$.
• Figure 3: Design domain, boundary conditions, and loading for the compliance minimization design problem. We take $L=1$ and use the symmetry of the problem to only simulate a quarter of the domain.
• Figure 4: Results of the density-based topology optimization with $100\%$ acceleration using the surrogate model. We build the linear surrogate in a $2$-dimensional subspace. This dramatic reduction in the dimensionality (from $n=13824$ to $r=2$) keeps surrogate evaluations lightweight once constructed, as each update is carried out in a rank-$r$ subspace.
• Figure 5: Results of the density-based topology optimization with $250\%$ acceleration using the surrogate model. The learned model for the linear optimization dynamics is built in a $2$-dimensional subspace. The surrogate model again closely tracks the reference solution obtained with gradient descent.

Theorems & Definitions (1)

• Remark 1