Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differentiable SVD based on Moore-Penrose Pseudoinverse for Inverse Imaging Problems

Yinghao Zhang, Yue Hu

TL;DR

This work shows that the non-differentiability of SVD is essentially due to an underdetermined system of linear equations arising in the derivation process, and utilizes the Moore-Penrose pseudoinverse to solve the system, thereby proposing a differentiable SVD.

Abstract

Low-rank regularization-based deep unrolling networks have achieved remarkable success in various inverse imaging problems (IIPs). However, the singular value decomposition (SVD) is non-differentiable when duplicated singular values occur, leading to severe numerical instability during training. In this paper, we propose a differentiable SVD based on the Moore-Penrose pseudoinverse to address this issue. To the best of our knowledge, this is the first work to provide a comprehensive analysis of the differentiability of the trivial SVD. Specifically, we show that the non-differentiability of SVD is essentially due to an underdetermined system of linear equations arising in the derivation process. We utilize the Moore-Penrose pseudoinverse to solve the system, thereby proposing a differentiable SVD. A numerical stability analysis in the context of IIPs is provided. Experimental results in color image compressed sensing and dynamic MRI reconstruction show that our proposed differentiable SVD can effectively address the numerical instability issue while ensuring computational precision. Code is available at https://github.com/yhao-z/SVD-inv.

Differentiable SVD based on Moore-Penrose Pseudoinverse for Inverse Imaging Problems

TL;DR

This work shows that the non-differentiability of SVD is essentially due to an underdetermined system of linear equations arising in the derivation process, and utilizes the Moore-Penrose pseudoinverse to solve the system, thereby proposing a differentiable SVD.

Abstract

Low-rank regularization-based deep unrolling networks have achieved remarkable success in various inverse imaging problems (IIPs). However, the singular value decomposition (SVD) is non-differentiable when duplicated singular values occur, leading to severe numerical instability during training. In this paper, we propose a differentiable SVD based on the Moore-Penrose pseudoinverse to address this issue. To the best of our knowledge, this is the first work to provide a comprehensive analysis of the differentiability of the trivial SVD. Specifically, we show that the non-differentiability of SVD is essentially due to an underdetermined system of linear equations arising in the derivation process. We utilize the Moore-Penrose pseudoinverse to solve the system, thereby proposing a differentiable SVD. A numerical stability analysis in the context of IIPs is provided. Experimental results in color image compressed sensing and dynamic MRI reconstruction show that our proposed differentiable SVD can effectively address the numerical instability issue while ensuring computational precision. Code is available at https://github.com/yhao-z/SVD-inv.

Paper Structure

This paper contains 15 sections, 3 theorems, 39 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. DUNs with Low-Rank Regularization for IIPs
  4. SVD Gradient
  5. Differentiable SVD based on Moore-Penrose Pseudoinverse
  6. The gradient of the proposed SVD-inv
  7. Numerical stability analysis in the context of IIPs
  8. SVD-inv for positive semi-definite matrices
  9. Experiments
  10. Efficacy of SVD-inv
  11. Applications in IIPs
  12. Color image compressive sensing
  13. Dynamic MRI reconstruction
  14. Results
  15. Discussion and Conclusion

Key Result

Theorem 3.1

Given the SVD $A=USV^H$ where $A\in\mathbb{C}^{m\times n}$, $U\in\mathbb{C}^{m\times k}$, $V\in\mathbb{C}^{n\times k}$, $S\in\mathbb{R}^{k\times k}$ and $\operatorname{rank}(A)<k<\min(m,n)$, the following relationships hold, where $I_k$ represents the $k \times k$ identity matrix, $\odot$ denotes the Hadamard product, $F_{ij}=$ , $T_{ij}=$, and $S^{-1}=$.

Figures (5)

  • Figure 1: The four split for $k \times k$ size.
  • Figure 2: The experimental settings for evaluating the efficacy of SVD-inv.
  • Figure 3: The framework of the used DUN.
  • Figure 4: The benchmark color images as the test set.
  • Figure 5: The sampling masks and the example images of OCMR. The sampling masks from left to right are the pseudo-radial sampling, variable density sampling and VISTA mask, respectively.

Theorems & Definitions (6)

  • Theorem 3.1
  • proof
  • Theorem 3.2
  • proof
  • Lemma 3.3
  • proof