Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Subspace embedding with random Khatri-Rao products and its application to eigensolvers

Zvonimir Bujanović, Luka Grubišić, Daniel Kressner, Hei Yin Lam

TL;DR

The paper addresses the challenge of efficiently computing parts of the spectrum for large-scale, often Kronecker-structured, matrices by exploiting randomized embeddings. It introduces random Khatri--Rao product sketches as structure-preserving dimension reductions and proves oblivious subspace embedding properties with explicit sample complexities for Gaussian KR matrices. It then demonstrates two algorithmic benefits: (i) within contour-integral eigensolvers, KR sketches transform shifted linear systems into Sylvester equations that leverage Kronecker structure for speed and memory efficiency, and (ii) a low-rank variant of LOBPCG stores iterates in a block low-rank format to exploit Kronecker structure and cap ranks via truncation, enabling scalable eigenvalue computations. Numerical experiments on Kronecker-structured discretizations (e.g., Schrödinger-type operators) show that KR-based methods achieve comparable eigen-subspace accuracy to Gaussian sketches while offering notable memory and time savings, especially when the Sylvester-based solver and low-rank representations are employed. Overall, the work provides both rigorous probabilistic guarantees and practical algorithms for fast, structure-aware eigensolvers in Kronecker-structured settings.

Abstract

Various iterative eigenvalue solvers have been developed to compute parts of the spectrum for a large sparse matrix, including the power method, Krylov subspace methods, contour integral methods, and preconditioned solvers such as the so called LOBPCG method. All of these solvers rely on random matrices to determine, e.g., starting vectors that have, with high probability, a non-negligible overlap with the eigenvectors of interest. For this purpose, a safe and common choice are unstructured Gaussian random matrices. In this work, we investigate the use of random Khatri-Rao products in eigenvalue solvers. On the one hand, we establish a novel subspace embedding property that provides theoretical justification for the use of such structured random matrices. On the other hand, we highlight the potential algorithmic benefits when solving eigenvalue problems with Kronecker product structure, as they arise frequently from the discretization of eigenvalue problems for differential operators on tensor product domains. In particular, we consider the use of random Khatri-Rao products within a contour integral method and LOBPCG. Numerical experiments indicate that the gains for the contour integral method strongly depend on the ability to efficiently and accurately solve (shifted) matrix equations with low-rank right-hand side. The flexibility of LOBPCG to directly employ preconditioners makes it easier to benefit from Khatri-Rao product structure, at the expense of having less theoretical justification.

Subspace embedding with random Khatri-Rao products and its application to eigensolvers

TL;DR

The paper addresses the challenge of efficiently computing parts of the spectrum for large-scale, often Kronecker-structured, matrices by exploiting randomized embeddings. It introduces random Khatri--Rao product sketches as structure-preserving dimension reductions and proves oblivious subspace embedding properties with explicit sample complexities for Gaussian KR matrices. It then demonstrates two algorithmic benefits: (i) within contour-integral eigensolvers, KR sketches transform shifted linear systems into Sylvester equations that leverage Kronecker structure for speed and memory efficiency, and (ii) a low-rank variant of LOBPCG stores iterates in a block low-rank format to exploit Kronecker structure and cap ranks via truncation, enabling scalable eigenvalue computations. Numerical experiments on Kronecker-structured discretizations (e.g., Schrödinger-type operators) show that KR-based methods achieve comparable eigen-subspace accuracy to Gaussian sketches while offering notable memory and time savings, especially when the Sylvester-based solver and low-rank representations are employed. Overall, the work provides both rigorous probabilistic guarantees and practical algorithms for fast, structure-aware eigensolvers in Kronecker-structured settings.

Abstract

Various iterative eigenvalue solvers have been developed to compute parts of the spectrum for a large sparse matrix, including the power method, Krylov subspace methods, contour integral methods, and preconditioned solvers such as the so called LOBPCG method. All of these solvers rely on random matrices to determine, e.g., starting vectors that have, with high probability, a non-negligible overlap with the eigenvectors of interest. For this purpose, a safe and common choice are unstructured Gaussian random matrices. In this work, we investigate the use of random Khatri-Rao products in eigenvalue solvers. On the one hand, we establish a novel subspace embedding property that provides theoretical justification for the use of such structured random matrices. On the other hand, we highlight the potential algorithmic benefits when solving eigenvalue problems with Kronecker product structure, as they arise frequently from the discretization of eigenvalue problems for differential operators on tensor product domains. In particular, we consider the use of random Khatri-Rao products within a contour integral method and LOBPCG. Numerical experiments indicate that the gains for the contour integral method strongly depend on the ability to efficiently and accurately solve (shifted) matrix equations with low-rank right-hand side. The flexibility of LOBPCG to directly employ preconditioners makes it easier to benefit from Khatri-Rao product structure, at the expense of having less theoretical justification.
Paper Structure (24 sections, 11 theorems, 63 equations, 7 figures, 3 algorithms)

This paper contains 24 sections, 11 theorems, 63 equations, 7 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Random Khatri--Rao matrices.
  3. Eigenvalue solvers with random Khatri--Rao matrices.
  4. Related work.
  5. Structure of the paper.
  6. Oblivious subspace embedding with random Khatri--Rao products
  7. JL moment property
  8. Subspace embedding property for Khatri--Rao product of Gaussian matrices
  9. Application of random Khatri--Rao matrices to contour integral eigensolvers
  10. Efficient implementation
  11. Low-rank variant of LOBPCG
  12. Exploiting low-rank and Kronecker structure
  13. Definition of block low-rank format.
  14. Application of operators.
  15. Addition.
  16. ...and 9 more sections

Key Result

Theorem 3

Let $\varepsilon\in (0,1]$, $\delta\in (0,e^{-8}]$ and $n=\tilde{n}\hat{n}$. Choose $\Omega=\frac{1}{\sqrt{\ell}}(\tilde{\Omega} \odot \hat{\Omega})\in \mathbb{R}^{n\times \ell}$ with independent Gaussian random matrices $\tilde{\Omega}\in \mathbb{R}^{\tilde{n}\times \ell}$ and $\hat{\Omega}\in \mat

Figures (7)

  • Figure 1: The smallest number of samples $\ell$ in $\Omega \in \mathbb{R}^{400 \times \ell}$ such that the empirical probability of the event $\|(\Omega^T U)^\dagger\|_2\geq 5$ is smaller than $1/50$. Here $U \in \mathbb{R}^{400 \times k}$ has orthonormal columns, $k=4,\ldots, 20$. Left: $U$ is randomly generated; right: $U$ has rank-one vectors as columns.
  • Figure 2: Statistics for $\|(\Omega^T U)^\dagger\|_2$ in $1000$ trials with a random matrix $\Omega \in \mathbb{R}^{400 \times \ell}$. Here $U \in \mathbb{R}^{400 \times 8}$ is a fixed matrix with orthonormal columns. Left: $U$ is randomly generated; right: $U$ has rank-one vectors as columns.
  • Figure 3: Properties of the eigenvalue problem arising from a discretized Schrödinger equation with potential $V(x, y) = (x^2 + y^2 - xy)/2$.
  • Figure 4: Residuals and absolute errors of Algorithm \ref{['alg:lobpcg']} (LOBPCG with low-rank truncation) applied to Example \ref{['ex:lobpcg-sum-of-squares']}.
  • Figure 5: Residuals and absolute errors of Algorithm \ref{['alg:lobpcg']} (LOBPCG with low-rank truncation) applied to Example \ref{['ex:lobpcg-gaussian']}.
  • ...and 2 more figures

Theorems & Definitions (31)

  • Definition 1
  • Definition 2
  • Theorem 3
  • Theorem 4
  • proof
  • Theorem 5
  • proof
  • Example 6
  • Theorem 7
  • proof
  • ...and 21 more