Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Preconditioned log-determinant approximation: one Gaussian probe vector is almost always enough

Alice Cortinovis, Daniele Toni

TL;DR

This work tackles the computational challenge of estimating the regularized log-determinant ${\rm log\,det}(A+I)$ for SPSD matrices using only matrix-vector products. It introduces a Nyström-based preconditioner to reduce variance and conditioning, and employs stochastic Lanczos quadrature with a single Gaussian probe to estimate the residual trace, achieving strong performance when spectral decay is moderate. A complementary detective strategy monitors preconditioner quality and adaptively switches budgeting between the Nyström approximation and SLQ to maintain accuracy. Theoretical bounds for the one-sample Nyström approach are derived, and extensive numerical experiments on synthetic and kernel matrices demonstrate competitive accuracy and scalability compared to existing methods. The approach offers practical, efficient log-determinant estimation suitable for large-scale problems in machine learning and statistics, including Gaussian processes and kernelized models.

Abstract

We present randomized algorithms for estimating the log-determinant of regularized symmetric positive semi-definite matrices. The algorithms access the matrix only through matrix vector products, and are based on the introduction of a preconditioner and stochastic trace estimator. We claim that preconditioning as much as we can and making a rough estimate of the residual part with a small budget achieves a small error in most of the cases. We choose a Nyström preconditioner and estimate the residual using only one sample of stochastic Lanczos quadrature (SLQ). We analyze the performance of this strategy from a theoretical and practical viewpoint. We also present an algorithm that, at almost no additional cost, detects whether the proposed strategy is not the most effective, in which case it uses more samples for the SLQ part. Numerical examples on several test matrices show that our proposed methods are competitive with existing algorithms.

Preconditioned log-determinant approximation: one Gaussian probe vector is almost always enough

TL;DR

This work tackles the computational challenge of estimating the regularized log-determinant for SPSD matrices using only matrix-vector products. It introduces a Nyström-based preconditioner to reduce variance and conditioning, and employs stochastic Lanczos quadrature with a single Gaussian probe to estimate the residual trace, achieving strong performance when spectral decay is moderate. A complementary detective strategy monitors preconditioner quality and adaptively switches budgeting between the Nyström approximation and SLQ to maintain accuracy. Theoretical bounds for the one-sample Nyström approach are derived, and extensive numerical experiments on synthetic and kernel matrices demonstrate competitive accuracy and scalability compared to existing methods. The approach offers practical, efficient log-determinant estimation suitable for large-scale problems in machine learning and statistics, including Gaussian processes and kernelized models.

Abstract

We present randomized algorithms for estimating the log-determinant of regularized symmetric positive semi-definite matrices. The algorithms access the matrix only through matrix vector products, and are based on the introduction of a preconditioner and stochastic trace estimator. We claim that preconditioning as much as we can and making a rough estimate of the residual part with a small budget achieves a small error in most of the cases. We choose a Nyström preconditioner and estimate the residual using only one sample of stochastic Lanczos quadrature (SLQ). We analyze the performance of this strategy from a theoretical and practical viewpoint. We also present an algorithm that, at almost no additional cost, detects whether the proposed strategy is not the most effective, in which case it uses more samples for the SLQ part. Numerical examples on several test matrices show that our proposed methods are competitive with existing algorithms.
Paper Structure (24 sections, 2 theorems, 55 equations, 8 figures, 1 algorithm)

This paper contains 24 sections, 2 theorems, 55 equations, 8 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Relation with existing literature
  4. Outline
  5. Analysis of the proposed strategy in the idealized case
  6. Using the Nyström preconditioner
  7. Analysis of the one-sample strategy
  8. Comparison with low-rank and alpha-rank
  9. Inexact quadratic forms with Lanczos method
  10. Error control and a "detective" version
  11. Evaluating the error of the Nyström approximation
  12. Choice of beta
  13. Condition for switching the strategy
  14. Updating the Nyström approximation
  15. Numerical Experiments
  16. ...and 9 more sections

Key Result

Lemma 3.1

\newlabellemma:NysMk_fro0 For any SPSD matrix $A \in \mathbb{R}^{n \times n}$, for any $k\ge0$ and $p \ge 2$ such that $k+p=\ell$, with $\ell \ge 4$, we have where $\bar{\Lambda}_k \in \mathbb R^{(n-k)\times (n-k)}$ is the diagonal matrix that contains the trailing $n-k$ eigenvalues of $A$.

Figures (8)

  • Figure 1: Square root of the variance \ref{['eq:experrideal-one']} for the \ref{['eq:onesample-idealized']} strategy divided by ${\rm trace}\log(A+I)$, error \ref{['eq:errTrunc_ideal']} for the \ref{['eq:lowrank']} strategy divided by ${\rm trace}\log(A+I)$ and square root of the variance \ref{['eq:errMix_ideal']} divided by ${\rm trace}\log(A+I)$ for \ref{['eq:mixed_strategy']} strategies for $\alpha=0.1,\ldots,0.9$, represented by thin lines, shading from purple ($\alpha=0.1$) to blue ($\alpha=0.9$) considering $m=10$ and the matrices $A\in \mathbb R^{4000\times 4000}$ described in Section \ref{['subsec:matrices']}.
  • Figure 1: Spectra of the scaled matrices $A\in \mathbb R^{4000\times 4000}$ described in Section \ref{['subsec:matrices']}; note that the $y$-axis has a logarithmic scale.
  • Figure 2: Square root of the upper bound \ref{['eq:exp_logdet_true_fro']} and computed error, on average, for the one-sample strategy \ref{['eq:onesample-nystrom']}, upper bound \ref{['eq:err:Trunc_Nys_leading']} and computed error, on average, for the low-rank strategy \ref{['eq:lowrank']}, considering a Nyström approximation of dimension $\ell=100,\ldots,1000$ and $m=50$ steps of the Lanczos method. Each bound is the optimal among all the combination of $k$ and $p$ with fixed sum, as described in Section \ref{['subsec:bounds']}.
  • Figure 3: Square root of the upper bound \ref{['eq:exp_logdet_true_fro']} and computed error, on average, for the one-sample strategy \ref{['eq:onesample-nystrom']}, upper bound \ref{['eq:err:Trunc_Nys_leading']} and computed error, on average, for the low-rank strategy \ref{['eq:lowrank']}, considering a Nyström approximation of dimension $\ell=100,\ldots,1000$ and $m=10$ steps of the Lanczos method. Each bound is the optimal among all the combination of $k$ and $p$ with fixed sum, as described in Section \ref{['subsec:bounds']}.
  • Figure 4: Plot of the error of the Lanczos method \ref{['eq:lanczos']}, averaging on $100$ Gaussian samples, divided by the target quantity ${\rm trace}\log(A+I)$ after $m=10$ and $m=50$ steps and comparison with the error of the trace estimator \ref{['eq:baderror']}, averaging on the same Gaussian samples and dividing by the target quantity ${\rm trace}\log(A+I)$, considering an approximation of dimension $\ell=100,\ldots,1000$ as described in Section \ref{['subsec:expsLanczos']}
  • ...and 3 more figures

Theorems & Definitions (5)

  • Lemma 3.1
  • Proof 1
  • Theorem 3.2
  • Remark 5.1
  • Remark 5.2