Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Greedy construction of quadratic manifolds for nonlinear dimensionality reduction and nonlinear model reduction

Paul Schwerdtner, Benjamin Peherstorfer

TL;DR

This work tackles nonlinear dimensionality reduction by augmenting linear encodings with quadratic corrections. It introduces a greedy subspace construction that selects basis vectors from among the first $m$ left-singular vectors of the data, rather than relying solely on the leading $r$ vectors, to maximize the information captured by the polynomial feature map used in the correction term. The approach yields orders-of-magnitude improvements in accuracy over PCA-based linear methods and is scalable to data with millions of dimensions, as demonstrated on several physics-inspired datasets. The method combines a tractable encoder $f_{oldsymbol{V}}$, a quadratic correction via $H(oldsymbol{s}_r)=oldsymbol{W} h(oldsymbol{s}_r)$, and efficient SVD-based acceleration, with open-source code available for practitioners.

Abstract

Dimensionality reduction on quadratic manifolds augments linear approximations with quadratic correction terms. Previous works rely on linear approximations given by projections onto the first few leading principal components of the training data; however, linear approximations in subspaces spanned by the leading principal components alone can miss information that are necessary for the quadratic correction terms to be efficient. In this work, we propose a greedy method that constructs subspaces from leading as well as later principal components so that the corresponding linear approximations can be corrected most efficiently with quadratic terms. Properties of the greedily constructed manifolds allow applying linear algebra reformulations so that the greedy method scales to data points with millions of dimensions. Numerical experiments demonstrate that an orders of magnitude higher accuracy is achieved with the greedily constructed quadratic manifolds compared to manifolds that are based on the leading principal components alone.

Greedy construction of quadratic manifolds for nonlinear dimensionality reduction and nonlinear model reduction

TL;DR

This work tackles nonlinear dimensionality reduction by augmenting linear encodings with quadratic corrections. It introduces a greedy subspace construction that selects basis vectors from among the first left-singular vectors of the data, rather than relying solely on the leading vectors, to maximize the information captured by the polynomial feature map used in the correction term. The approach yields orders-of-magnitude improvements in accuracy over PCA-based linear methods and is scalable to data with millions of dimensions, as demonstrated on several physics-inspired datasets. The method combines a tractable encoder , a quadratic correction via , and efficient SVD-based acceleration, with open-source code available for practitioners.

Abstract

Dimensionality reduction on quadratic manifolds augments linear approximations with quadratic correction terms. Previous works rely on linear approximations given by projections onto the first few leading principal components of the training data; however, linear approximations in subspaces spanned by the leading principal components alone can miss information that are necessary for the quadratic correction terms to be efficient. In this work, we propose a greedy method that constructs subspaces from leading as well as later principal components so that the corresponding linear approximations can be corrected most efficiently with quadratic terms. Properties of the greedily constructed manifolds allow applying linear algebra reformulations so that the greedy method scales to data points with millions of dimensions. Numerical experiments demonstrate that an orders of magnitude higher accuracy is achieved with the greedily constructed quadratic manifolds compared to manifolds that are based on the leading principal components alone.
Paper Structure (31 sections, 1 theorem, 41 equations, 19 figures, 1 algorithm)

This paper contains 31 sections, 1 theorem, 41 equations, 19 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Linear approximations in subspaces
  4. Manifold approximations via nonlinear decoders
  5. Corrections that depend on encoded data points only
  6. Manifold approximations given by nonlinear corrections
  7. Correction terms via polynomial feature maps
  8. Problem formulation
  9. Greedy construction of quadratic manifolds
  10. Greedy selection strategy
  11. Accelerating repeated least-squares solves for efficient greedy selection
  12. Re-using the pre-computed SVD of the data matrix
  13. Reduced number of unknowns in least-squares problems
  14. Algorithm description
  15. Discussion
  16. ...and 16 more sections

Key Result

Proposition 1

Consider a manifold $\mathcal{M}_r(\mathbf{V}, H)$ as defined in eq:Prelim:ManifoldDef with matrix $\mathbf{V} \in \mathbb{R}^{{n} \times r}$ and dimension $r \leq {n}$. The correction map $H: \mathbb{R}^{r} \to \mathbb{R}^{{n}}$ is given via a feature map $h: \mathbb{R}^{r} \to \mathbb{R}^{{p}}$ as

Figures (19)

  • Figure 1: The plots show the data points given by the data matrix $\mathbf{S}^{(\mathrm{parabola})}$ and their respective approximation on one-dimensional quadratic manifolds. Constructing the quadratic manifold based on the leading $r = 1$ left-singular vector alone leads to poor approximations, as can be seen in the left plot. In contrast, greedily selecting the subspace $\mathcal{V}$ with the proposed approach leads to quadratic-manifold approximations that exactly represent the data points, see plot on the right.
  • Figure 2: Advecting wave: The proposed greedy approach achieves up to five orders of magnitude higher accuracy than using the leading $r$ left-singular vectors for the quadratic manifold construction. Additionally, the greedy approach incurs an orders of magnitude lower runtime than alternating minimization.
  • Figure 3: Advecting wave: Runtime and accuracy of the greedy method can be traded off by varying the number $m$ of left-singular vectors that are considered in each greedy iteration.
  • Figure 4: Advecting wave: Singular value decay of training data set and relative error comparison to lower bound and nonlinear encoding.
  • Figure 5: Advecting wave: The plots show the magnitude of the entries of the correlation matrix \ref{['eq:correlations_compute']}. It is necessary that the lifted encoded data points are correlated in the sense of \ref{['eq:correlations_compute']} with the coordinates corresponding the later left-singular vectors for the quadratic correction term to be effective; see Section \ref{['sec:discussion_correlation']}. Plot (a) shows that a basis matrix $\mathbf{V}$ using the first $r = 20$ left-singular vectors only lead to lifted encoded data points that are poorly correlated with the coordinates of projections onto later left-singular vectors (lower rows). In contrast, the left-singular vectors selected by the proposed greedy method achieve stronger correlation as shown in plot (b) and in alignment with the other results in this section.
  • ...and 14 more figures

Theorems & Definitions (3)

  • Remark 1
  • Proposition 1
  • proof