Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast algorithms for least square problems with Kronecker lower subsets

Osman Asif Malik, Yiming Xu, Nuojin Cheng, Stephen Becker, Alireza Doostan, Akil Narayan

TL;DR

The paper tackles the high computational cost of exact leverage score sampling for large-scale least squares problems when the design matrix is a lower column subset of a Kronecker product. It introduces an efficient algorithm that uses offline QR factorizations of the factor matrices and a two-step online sampling procedure, guaranteeing that the sampled rows follow the exact leverage-score distribution without forming the full matrix. The authors provide theoretical guarantees, including an equivalence between the proposed procedure and exact leverage-score sampling, and demonstrate practical gains through experiments on polynomial chaos expansions for a Duffing oscillator, the Ishigami function, and battery remaining useful life, showing improvements over uniform and approximate sampling. The work highlights the potential of exploiting Kronecker-structured lower-subset matrices to achieve accurate, scalable sketching for high-dimensional uncertainty quantification tasks, and it suggests future work on broader subset structures and automation of suitable permutations.

Abstract

While leverage score sampling provides powerful tools for approximating solutions to large least squares problems, the cost of computing exact scores and sampling often prohibits practical application. This paper addresses this challenge by developing a new and efficient algorithm for exact leverage score sampling applicable to matrices that are lower column subsets of Kronecker product matrices. We synthesize relevant approximation guarantees and detail the algorithm that specifically leverages this structural property for computational efficiency. Through numerical examples, we demonstrate that utilizing efficiently computed exact leverage scores via our methods significantly reduces approximation errors, as compared to established approximate leverage score sampling strategies when applied to this important class of structured matrices.

Fast algorithms for least square problems with Kronecker lower subsets

TL;DR

The paper tackles the high computational cost of exact leverage score sampling for large-scale least squares problems when the design matrix is a lower column subset of a Kronecker product. It introduces an efficient algorithm that uses offline QR factorizations of the factor matrices and a two-step online sampling procedure, guaranteeing that the sampled rows follow the exact leverage-score distribution without forming the full matrix. The authors provide theoretical guarantees, including an equivalence between the proposed procedure and exact leverage-score sampling, and demonstrate practical gains through experiments on polynomial chaos expansions for a Duffing oscillator, the Ishigami function, and battery remaining useful life, showing improvements over uniform and approximate sampling. The work highlights the potential of exploiting Kronecker-structured lower-subset matrices to achieve accurate, scalable sketching for high-dimensional uncertainty quantification tasks, and it suggests future work on broader subset structures and automation of suitable permutations.

Abstract

While leverage score sampling provides powerful tools for approximating solutions to large least squares problems, the cost of computing exact scores and sampling often prohibits practical application. This paper addresses this challenge by developing a new and efficient algorithm for exact leverage score sampling applicable to matrices that are lower column subsets of Kronecker product matrices. We synthesize relevant approximation guarantees and detail the algorithm that specifically leverages this structural property for computational efficiency. Through numerical examples, we demonstrate that utilizing efficiently computed exact leverage scores via our methods significantly reduces approximation errors, as compared to established approximate leverage score sampling strategies when applied to this important class of structured matrices.
Paper Structure (20 sections, 4 theorems, 36 equations, 4 figures, 1 algorithm)

This paper contains 20 sections, 4 theorems, 36 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Row-structured sketches
  5. Kronecker fast Johnson--Lindenstrauss transforms (KFJLTs)
  6. TensorSketch
  7. Recursive sketches
  8. Sampling-based sketches
  9. Preliminaries
  10. Leverage Score Sampling
  11. Kronecker Product Structure and Lower Column Subsets
  12. Our Contribution: Efficient Leverage Score Sampling on Kronecker Lower Subsets
  13. Sampling Algorithm
  14. Proofs
  15. Experiments
  16. ...and 5 more sections

Key Result

Proposition 2.2

Let $\bm{Q} \in \mathbb{R}^{M \times N}$ be any matrix whose columns form an orthonormal basis for the column space of $\bm{A}$ (i.e., $\textup{span}(\bm{Q}) = \textup{span}(\bm{A})$ and $\bm{Q}^\mathsf{T} \bm{Q} = \bm{I}_N$). Then the leverage score of the $m$-th row is the squared Euclidean norm o where $q_{mn}$ denotes the $(m, n)$-th entry of $\bm{Q}$ and $\bm{I}_N$ is the $N \times N$ identit

Figures (4)

  • Figure 1: Illustration of index sets transformable into lower subsets. The sets $\mathcal{J}$ (left) and $\widetilde{\mathcal{J}}$ (third from left) are not lower subsets. Applying suitable dimension-wise permutations $\pi$ and $\widetilde{\pi}$ yields the sets $\pi(\mathcal{J})$ (second from left) and $\widetilde{\pi}(\widetilde{\mathcal{J}})$ (right), which are lower subsets.
  • Figure 2: Empirical CDFs of the relative error in estimating the displacement $u(\bm{y},4)$ of the Duffing oscillator. Top row: Total degree (TD) PCE of order $k=9$ (left, optimal rel. error $2.6\times 10^{-2}$) and $k=12$ (right, optimal rel. error $2.9\times 10^{-3}$). Bottom row: Hyperbolic cross (HC) PCE of order $k=15$ (left, optimal rel. error $6.9\times 10^{-2}$) and $k=18$ (right, optimal rel. error $3.2\times 10^{-2}$). Results compare Uniform (blue), TP Leverage (orange), and Exact Leverage (green) sampling over 100 trials.
  • Figure 3: Empirical CDFs of the relative error in approximating the Ishigami function $f(\bm{y})$. Top row: Total degree (TD) PCE of order $k=7$ (left, optimal rel. error $7.0\times 10^{-3}$) and $k=9$ (right, optimal rel. error $9.5\times 10^{-4}$). Bottom row: Hyperbolic cross (HC) PCE of order $k=15$ (left, optimal rel. error $9.0\times 10^{-2}$) and $k=18$ (right, optimal rel. error $7.7\times 10^{-2}$). Results compare Uniform (blue), TP Leverage (orange), and Exact Leverage (green) sampling over 100 trials.
  • Figure 4: Empirical CDFs of the relative error in estimating the battery RUL using PCE of degree $k=3$. Top row: Total degree (TD) space with $M_d=4$ quadrature points per dimension (left, optimal rel. error $6.6\times 10^{-4}$) and $M_d=5$ (right, optimal rel. error $9.3\times 10^{-4}$). Bottom row: Hyperbolic cross (HC) space with $M_d=4$ (left, optimal rel. error $2.2\times 10^{-3}$) and $M_d=5$ (right, optimal rel. error $1.1\times 10^{-3}$). Results compare Uniform (blue), TP Leverage (orange), and Exact Leverage (green) sampling over 100 trials.

Theorems & Definitions (10)

  • Definition 2.1: Leverage Score
  • Proposition 2.2: Equation 4.4 in meyer2023near
  • Definition 2.3: Multi-Index Set
  • Definition 2.4: Lower Subset
  • Proposition 3.1: Two-Step Univariate Sampling Equivalence
  • Theorem 3.2: Two-Step Multivariate Sampling Equivalence
  • proof : Proof of Proposition \ref{['prop:equavalence']}
  • Proposition 3.3
  • proof : Proof of Proposition \ref{['prop:Q-lower']}
  • proof : Proof of Theorem \ref{['thm:equivalence']}