Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerating Legacy Numerical Solvers by Non-intrusive Gradient-based Meta-solving

Sohei Arisaka, Qianxiao Li

TL;DR

This work tackles the challenge of accelerating legacy, non-differentiable numerical solvers by introducing NI-GBMS, a non-intrusive gradient-based meta-solving framework that jointly trains neural meta-solvers with legacy solvers using a forward-gradient estimator. A surrogate model and a control variate are employed to produce unbiased, low-variance gradient estimates, enabling end-to-end training without modifying existing code. Theoretical convergence analysis and numerical experiments demonstrate faster training and substantial reductions in solver iterations, including applications to Poisson, biharmonic, and linear elasticity problems in PETSc and FEniCS. The approach broadens the applicability of neural-guided solver optimization to real-world, non-differentiable software, offering practical speedups for large-scale scientific computing tasks.

Abstract

Scientific computing is an essential tool for scientific discovery and engineering design, and its computational cost is always a main concern in practice. To accelerate scientific computing, it is a promising approach to use machine learning (especially meta-learning) techniques for selecting hyperparameters of traditional numerical methods. There have been numerous proposals to this direction, but many of them require automatic-differentiable numerical methods. However, in reality, many practical applications still depend on well-established but non-automatic-differentiable legacy codes, which prevents practitioners from applying the state-of-the-art research to their own problems. To resolve this problem, we propose a non-intrusive methodology with a novel gradient estimation technique to combine machine learning and legacy numerical codes without any modification. We theoretically and numerically show the advantage of the proposed method over other baselines and present applications of accelerating established non-automatic-differentiable numerical solvers implemented in PETSc, a widely used open-source numerical software library.

Accelerating Legacy Numerical Solvers by Non-intrusive Gradient-based Meta-solving

TL;DR

This work tackles the challenge of accelerating legacy, non-differentiable numerical solvers by introducing NI-GBMS, a non-intrusive gradient-based meta-solving framework that jointly trains neural meta-solvers with legacy solvers using a forward-gradient estimator. A surrogate model and a control variate are employed to produce unbiased, low-variance gradient estimates, enabling end-to-end training without modifying existing code. Theoretical convergence analysis and numerical experiments demonstrate faster training and substantial reductions in solver iterations, including applications to Poisson, biharmonic, and linear elasticity problems in PETSc and FEniCS. The approach broadens the applicability of neural-guided solver optimization to real-world, non-differentiable software, offering practical speedups for large-scale scientific computing tasks.

Abstract

Scientific computing is an essential tool for scientific discovery and engineering design, and its computational cost is always a main concern in practice. To accelerate scientific computing, it is a promising approach to use machine learning (especially meta-learning) techniques for selecting hyperparameters of traditional numerical methods. There have been numerous proposals to this direction, but many of them require automatic-differentiable numerical methods. However, in reality, many practical applications still depend on well-established but non-automatic-differentiable legacy codes, which prevents practitioners from applying the state-of-the-art research to their own problems. To resolve this problem, we propose a non-intrusive methodology with a novel gradient estimation technique to combine machine learning and legacy numerical codes without any modification. We theoretically and numerically show the advantage of the proposed method over other baselines and present applications of accelerating established non-automatic-differentiable numerical solvers implemented in PETSc, a widely used open-source numerical software library.
Paper Structure (40 sections, 5 theorems, 31 equations, 4 figures, 5 tables)

This paper contains 40 sections, 5 theorems, 31 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. ML-enhanced Simulation
  4. Forward Gradient
  5. Black-box Functions and Neural Networks
  6. Method: Non-intrusive Gradient-based Meta-solving
  7. Analysis
  8. Theoretical Analysis
  9. Numerical Experiments
  10. Application: Learning Parameters of Legacy Numerical Solvers
  11. Toy Example: Poisson Equation
  12. Problem setting
  13. Baselines
  14. Results
  15. Advanced Examples
  16. ...and 25 more sections

Key Result

Theorem 3.4

If ${\textnormal{v}}_i$'s are independent and have zero mean and unit variance, then ${\bm{h}}_{\mathbf{v}} ({\bm{\theta}})$ is an unbiased estimator of $\nabla f({\bm{\theta}})$, i.e. Furtheremore, if ${\textnormal{v}}_i$'s are independent Rademacher variables, then the mean squared deviation of ${\bm{h}}_{\mathbf{v}}({\bm{\theta}})$ is minimized and equal to

Figures (4)

  • Figure 1: The overall architecture of non-intrusive gradient-based meta-solving
  • Figure 2: The convergence plot and cosine similarity (Sphere function $d=128$). The shadowed area represents a range from 10% to 90%.
  • Figure 3: The learning curves of the case $f_{\mathrm{MG}}$. They are trained using a same random seed and hyperparameters except for the gradient estimator.
  • Figure 4: Convergence plots. The horizontal axis is the number of Adam steps, and the vertical axis is the objective value.

Theorems & Definitions (13)

  • Definition 3.1: Meta-solving problem Arisaka2023-aw
  • Definition 3.2: Forward gradient Belouze2022-km
  • Definition 3.3: Control variate forward gradient
  • Theorem 3.4: Improvement by ${\bm{h}}_{\mathbf{v}}$
  • Theorem 4.1: Convergence
  • proof : Proof of \ref{['thm:property']}
  • proof : Proof of \ref{['thm:convergence']}
  • Lemma 1.1
  • proof
  • Lemma 1.2: Polyak1963-ys
  • ...and 3 more