Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning incomplete factorization preconditioners for GMRES

Paul Häusner, Aleix Nieto Juscafresa, Jens Sjölund

TL;DR

The paper tackles accelerating GMRES for sparse linear systems by learning to predict a sparse LU factorization with a graph neural network, ensuring invertibility and sparsity via a novel activation and ILU(0)-style constraints. It introduces multiple spectrum-aware losses that guide the learned preconditioner toward favorable singular-value distributions, using Hutchinson’s estimator to approximate spectral extremes. Experiments on synthetic Poisson-based problems show that carefully chosen losses reduce GMRES iterations and improve convergence similarly to classical ILU methods, while enabling offline training and fast online inference. The work demonstrates that data-driven preconditioners can complement traditional numerical algorithms, with future directions including broader problem classes and hybrid strategies with classical preconditioners.

Abstract

Incomplete LU factorizations of sparse matrices are widely used as preconditioners in Krylov subspace methods to speed up solving linear systems. Unfortunately, computing the preconditioner itself can be time-consuming and sensitive to hyper-parameters. Instead, we replace the hand-engineered algorithm with a graph neural network that is trained to approximate the matrix factorization directly. To apply the output of the neural network as a preconditioner, we propose an output activation function that guarantees that the predicted factorization is invertible. Further, applying a graph neural network architecture allows us to ensure that the output itself is sparse which is desirable from a computational standpoint. We theoretically analyze and empirically evaluate different loss functions to train the learned preconditioners and show their effectiveness in decreasing the number of GMRES iterations and improving the spectral properties on synthetic data. The code is available at https://github.com/paulhausner/neural-incomplete-factorization.

Learning incomplete factorization preconditioners for GMRES

TL;DR

The paper tackles accelerating GMRES for sparse linear systems by learning to predict a sparse LU factorization with a graph neural network, ensuring invertibility and sparsity via a novel activation and ILU(0)-style constraints. It introduces multiple spectrum-aware losses that guide the learned preconditioner toward favorable singular-value distributions, using Hutchinson’s estimator to approximate spectral extremes. Experiments on synthetic Poisson-based problems show that carefully chosen losses reduce GMRES iterations and improve convergence similarly to classical ILU methods, while enabling offline training and fast online inference. The work demonstrates that data-driven preconditioners can complement traditional numerical algorithms, with future directions including broader problem classes and hybrid strategies with classical preconditioners.

Abstract

Incomplete LU factorizations of sparse matrices are widely used as preconditioners in Krylov subspace methods to speed up solving linear systems. Unfortunately, computing the preconditioner itself can be time-consuming and sensitive to hyper-parameters. Instead, we replace the hand-engineered algorithm with a graph neural network that is trained to approximate the matrix factorization directly. To apply the output of the neural network as a preconditioner, we propose an output activation function that guarantees that the predicted factorization is invertible. Further, applying a graph neural network architecture allows us to ensure that the output itself is sparse which is desirable from a computational standpoint. We theoretically analyze and empirically evaluate different loss functions to train the learned preconditioners and show their effectiveness in decreasing the number of GMRES iterations and improving the spectral properties on synthetic data. The code is available at https://github.com/paulhausner/neural-incomplete-factorization.
Paper Structure (18 sections, 2 theorems, 19 equations, 6 figures, 2 tables, 3 algorithms)

This paper contains 18 sections, 2 theorems, 19 equations, 6 figures, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. GMRES algorithm
  4. Graph neural networks
  5. Method
  6. Network architecture
  7. Model training
  8. Experiments & Results
  9. Discussion & Conclusion
  10. GMRES algorithm
  11. Implementation details
  12. Dataset
  13. Network architecture
  14. Baselines
  15. Proofs
  16. ...and 3 more sections

Key Result

Lemma 1

The largest singular value of the matrix ${\bm{A}} {\bm{P}}^{-1}$ is upper bounded by the Frobenius norm: $\sigma_{\max}({\bm{A}}{\bm{P}}^{-1}) \leq \epsilon^{-1} \;\| {\bm{A}} - {\bm{P}} \|_F + 1$.

Figures (6)

  • Figure 1: Non-symmetric matrix ${\bm{A}}$ (left) and the corresponding sparse Coates graph representation (right). Additionally to the non-zero elements in the matrix, the graph has been modified by adding edges for all missing diagonal elements, corresponding to self-loops.
  • Figure 2: Plot of the activation function (\ref{['eq:activationfunction']}) for $\epsilon=0.1$ and the corresponding relaxation (\ref{['eq:activationrelaxed']}).
  • Figure 3: Descending log-scale plot of ordered singular values of a single problem instance for learned preconditioners derived from the different loss functions.
  • Figure B.1: Normalized training loss using different loss functions from Section \ref{['sec:method']}.
  • Figure D.1: Descending log-scale plot of ordered singular values of single problem instance for different baseline preconditioners.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Lemma 1
  • Lemma 2