Stability of the Lanczos Method for Matrix Function Approximation

TL;DR

This paper proves that finite precision Lanczos essentially matches the exact arithmetic guarantee if computations use roughly $\log(nC\|A\|)$ bits of precision, and raises the question of if convergence in less than $poly(\kappa)$ iterations can be expected in finite precision, even for matrices with clustered, skewed, or otherwise favorable eigenvalue distributions.

Abstract

The ubiquitous Lanczos method can approximate $f(A)x$ for any symmetric $n \times n$ matrix $A$, vector $x$, and function $f$. In exact arithmetic, the method's error after $k$ iterations is bounded by the error of the best degree-$k$ polynomial uniformly approximating $f(x)$ on the range $[λ_{min}(A), λ_{max}(A)]$. However, despite decades of work, it has been unclear if this powerful guarantee holds in finite precision. We resolve this problem, proving that when $\max_{x \in [λ_{min}, λ_{max}]}|f(x)| \le C$, Lanczos essentially matches the exact arithmetic guarantee if computations use roughly $\log(nC\|A\|)$ bits of precision. Our proof extends work of Druskin and Knizhnerman [DK91], leveraging the stability of the classic Chebyshev recurrence to bound the stability of any polynomial approximating $f(x)$. We also study the special case of $f(A) = A^{-1}$, where stronger guarantees hold. In exact arithmetic Lanczos performs as well as the best polynomial approximating $1/x$ at each of $A$'s eigenvalues, rather than on the full eigenvalue range. In seminal work, Greenbaum gives an approach to extending this bound to finite precision: she proves that finite precision Lanczos and the related CG method match any polynomial approximating $1/x$ in a tiny range around each eigenvalue [Gre89]. For $A^{-1}$, this bound appears stronger than ours. However, we exhibit matrices with condition number $κ$ where exact arithmetic Lanczos converges in $polylog(κ)$ iterations, but Greenbaum's bound predicts $Ω(κ^{1/5})$ iterations. It thus cannot offer significant improvement over the $O(κ^{1/2})$ bound achievable via our result. Our analysis raises the question of if convergence in less than $poly(κ)$ iterations can be expected in finite precision, even for matrices with clustered, skewed, or otherwise favorable eigenvalue distributions.

Figures (2)

• Figure 1: A sample "hard" distribution of eigenvalues with $z = 4$. The width of each range $\mathcal{R}_{i,j}$ is over-exaggerated for illustration -- in reality each interval has width $2\eta$, where $\eta \leq \frac{1}{4z\kappa}$.
• Figure 2: Plot of $\ln|1 - x/r|$ for $r = 1/10$. Proving that \ref{['cont_lower_bound_main_num_only']} cannot be too small for any root $r$ requires lower bounding a weighted integral of this function over $\mathcal{R} \subset [1/\kappa,1+\eta]$.

Theorems & Definitions (49)

• Theorem 1: Function Approximation via Lanczos in Finite Arithmetic
• Theorem 2: Conjugate Gradient in Finite Arithmetic greenbaum1989behavior
• Theorem 3: Stable Approximating Polynomial Lower Bound
• Corollary 4
• Definition 1: Matrix Function
• Claim 5: Lanczos Output Guarantee
• Theorem 6: Approximate Application of Matrix Functions
• Lemma 7: Exact Application of Polynomials
• proof
• proof : Proof of Theorem \ref{['exact_lanczos_final_theorem']}