Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Note on the Stability of the Sherman-Morrison-Woodbury Formula

Linkai Ma, Christos Boutsikas, Mehrdad Ghadiri, Petros Drineas

Abstract

We study the numerical stability of the Sherman-Morrison-Woodbury (SMW) identity. Let $B = A + UV^T$ and assume $U$ and $V$ both have full-column rank. We explore error bounds for the SMW identity when we are only able to compute approximate inverses. For both forward and backward errors, we present upper bounds as a function of the two-norm error of the approximate inverses. We verify with numerical experiments that, in certain cases, our bounds accurately capture the behavior of the errors.

A Note on the Stability of the Sherman-Morrison-Woodbury Formula

Abstract

We study the numerical stability of the Sherman-Morrison-Woodbury (SMW) identity. Let and assume and both have full-column rank. We explore error bounds for the SMW identity when we are only able to compute approximate inverses. For both forward and backward errors, we present upper bounds as a function of the two-norm error of the approximate inverses. We verify with numerical experiments that, in certain cases, our bounds accurately capture the behavior of the errors.

Paper Structure

This paper contains 29 sections, 10 theorems, 99 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Notation
  4. Prior Work
  5. Improving the bound of ghadiri2023bit for the two-norm case
  6. Two lemmas from prior work
  7. Our Bounds
  8. Forward Error Bound
  9. Assumptions
  10. The bound
  11. Proof of Theorem \ref{['forward_error_thm']}
  12. Backward Error Bound
  13. Assumptions
  14. The bound
  15. Proof of Theorem \ref{['backward_thm']}
  16. ...and 14 more sections

Key Result

Lemma 2.1

Let $\mathbf{M}, \mathbf{N}$ be $n\times n$ invertible matrices and let $\|\mathbf{N}^{-1}\|_2 \leq \rho$ for some $\rho >1$. Assume $\|\mathbf{M}-\mathbf{N}\|_2 \leq \epsilon \leq 1/2\rho$, then $\|\mathbf{M}^{-1}\|_2 \leq 2 \rho$, and

Figures (4)

  • Figure 1: Rank-20 updates of the form $\mathbf{B} = \mathbf{A} + \mathbf{U} \mathbf{V}^{T}$, where $\mathbf{A} \in \mathbb{R}^{1000 \times 1000}$ is a standard Gaussian matrix. $\mathbf{U},\mathbf{V} \in \mathbb{R}^{1000\times 20}$ are standard Gaussian matrices scaled such that their two-norms (squared) are equal to $\lambda = .5\sigma_{\min}(\mathbf{A})$ (top) or $\lambda = .5 \sigma_{\max}(\mathbf{A})$ (bottom). The horizontal axis represents the values of $\epsilon_{1} = \epsilon_2 = \epsilon$. The blue dotted line represents the forward error (i.e. the left-hand side of eqn. (\ref{['forward_bound']})), while the green dotted line shows the respective bound (i.e. the right-hand side of eqn. (\ref{['forward_bound']})), and the orange dotted line represents the terms in Corollory \ref{['remark:forward']}, specifically the term $2\epsilon \|\mathbf{A}^{-1}\|_2 + 12 \epsilon$. The vertical red dashed line represents the maximum value of $\epsilon$ that satisfies the assumption of eqn. (\ref{['forward_bound_assumption']}) for the forward error bound. The top plot shows the small update case, while the bottom plot shows the large update case.
  • Figure 2: Rank-20 updates of the form $\mathbf{B} = \mathbf{A} + \mathbf{U} \mathbf{V}^{T}$, where $\mathbf{A} \in \mathbb{R}^{1000 \times 1000}$ is a standard Gaussian matrix. $\mathbf{U},\mathbf{V} \in \mathbb{R}^{1000\times 20}$ are standard Gaussian matrices scaled such that their two-norms (squared) are equal to $\lambda = \frac{1}{2}\sigma_{\min}(\mathbf{A})$ (top) or $\lambda = \frac{1}{2} \sigma_{\max}(\mathbf{A})$ (bottom). The horizontal axis represents the values of $\epsilon_{1} = \epsilon_2 = \epsilon$. The blue dotted line represents the backward error (i.e. the left-hand side of eqn. (\ref{['backward_fullbound']})), while the green dotted line shows the respective bound (i.e. the right-hand side of eqn. (\ref{['backward_fullbound']})), and the orange dotted line represents the terms in Corollary \ref{['remark:backward']}, i.e, $2 \|\mathbf{A}\|_2^2 \epsilon_1 + 8\epsilon_2$. The red dotted line represents the direct inversion backward error. Moreover, the vertical red dashed line, black dashed line, and blue dashed line represent the maximum value of $\epsilon_1 = \epsilon_2 = \epsilon$ that satisfies the assumptions in eqns. (\ref{['backward_thm_assumption1']}), (\ref{['epsilon2_small']}) and (\ref{['epsilon2_small_2']}) respectively for the backward error bound. The top plot shows the small update case, while the bottom plot shows the large update case.
  • Figure 3: Rank-20 updates of the form $\mathbf{B} = \mathbf{A} + \mathbf{U} \mathbf{V}^{T}$, where $\mathbf{A} \in \mathbb{R}^{1000 \times 1000},\mathbf{U}\in \mathbb{R}^{1000\times 20} \text{ and }\mathbf{V} \in \mathbb{R}^{1000\times 20}$ are manually constructed matrices such that $\lambda = 2\sigma_{\min}(\mathbf{A})$ (top) or $\lambda = 2 \sigma_{\max}(\mathbf{A})$ (bottom). We set $\epsilon_1 = \epsilon_2 = 10^{-3}$ for the top plot and $\epsilon_1 = \epsilon_2 = 10^{-10}$ for the bottom plot. The horizontal axis represents the values of $\alpha = \|\left(\mathbf{I} + \mathbf{V}^T \mathbf{A}^{-1} \mathbf{U}\right)^{-1}\|_2$. The blue dotted line represents the forward error (i.e. the left-hand side of eqn. (\ref{['forward_bound']})), while the green dotted line shows the respective bound (i.e. the right-hand side of eqn. (\ref{['forward_bound']})), and the orange dotted line represents the terms in eqn. (\ref{['simple_forward_bound_2']}), i.e, $2 \|\mathbf{A}^{-1}\|_2^2 \lambda^2 \alpha^2 \epsilon_1$. Moreover, the vertical red dashed line represents the maximum value of $\alpha$ that satisfies the assumption in eqn. (\ref{['forward_bound_assumption']}) for the forward error bound. The top plot shows the small update case, while the bottom plot shows the large update case.
  • Figure 4: Rank-20 updates of the form $\mathbf{B} = \mathbf{A} + \mathbf{U} \mathbf{V}^{T}$, where $\mathbf{A} \in \mathbb{R}^{1000 \times 1000},\mathbf{U}\in \mathbb{R}^{1000\times 20} \text{ and }\mathbf{V} \in \mathbb{R}^{1000\times 20}$ are manually constructed matrices such that $\lambda = 100\sigma_{\min}(\mathbf{A})$ (top) or $\lambda = 2 \sigma_{\max}(\mathbf{A})$ (bottom). We set $\epsilon_1 = \epsilon_2 = 10^{-6}$ for all the experiments. The horizontal axis represents the values of $\beta = \|{\mathbf{I} + \mathbf{V}^T \mathbf{A}^{-1} \mathbf{U}}\|_2$. The blue dotted line represents the backward error (i.e. the left-hand side of eqn. (\ref{['backward_fullbound']})), while the green dotted line shows the respective bound (i.e. the right-hand side of eqn. (\ref{['backward_fullbound']})), and the orange dotted line represents the terms in eqn. (\ref{['simple_forward_bound_2']}), i.e, $4\lambda \epsilon_2 \beta^2$. The top plot shows the small update case, while the bottom plot shows the large update case. For the top plot, we checked that all the 3 assumptions from Thm \ref{['backward_thm']}: eqns. (\ref{['backward_thm_assumption1']}), (\ref{['epsilon2_small']}) and (\ref{['epsilon2_small_2']}) are satisfied in all the experiments. For the bottom plot, we checked that both eqn. (\ref{['backward_thm_assumption1']}) and eqn. (\ref{['epsilon2_small']}) are satisfied in all the experiments. The assumption eqn. (\ref{['epsilon2_small_2']}) will be violated when $\beta$ becomes too large. The blue dashed line represents the maximum value of $\beta$ such that eqn. (\ref{['epsilon2_small_2']}) is satisfied.

Theorems & Definitions (11)

  • Lemma 2.1
  • Remark 1
  • Lemma 2.2
  • Theorem 2
  • Corollary 3
  • Corollary 4
  • Corollary 5
  • Theorem 6
  • Corollary 7
  • Corollary 8
  • ...and 1 more