Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Precise Performance of Linear Denoisers in the Proportional Regime

Reza Ghane, Danil Akhtiamov, Babak Hassibi

Abstract

In the present paper we study the performance of linear denoisers for noisy data of the form $\mathbf{x} + \mathbf{z}$, where $\mathbf{x} \in \mathbb{R}^d$ is the desired data with zero mean and unknown covariance $\mathbfΣ$, and $\mathbf{z} \sim \mathcal{N}(0, \mathbfΣ_{\mathbf{z}})$ is additive noise. Since the covariance $\mathbfΣ$ is not known, the standard Wiener filter cannot be employed for denoising. Instead we assume we are given samples $\mathbf{x}_1,\dots,\mathbf{x}_n \in \mathbb{R}^d$ from the true distribution. A standard approach would then be to estimate $\mathbfΣ$ from the samples and use it to construct an ``empirical" Wiener filter. However, in this paper, motivated by the denoising step in diffusion models, we take a different approach whereby we train a linear denoiser $\mathbf{W}$ from the data itself. In particular, we synthetically construct noisy samples $\hat{\mathbf{x}}_i$ of the data by injecting the samples with Gaussian noise with covariance $\mathbfΣ_1 \neq \mathbfΣ_{\mathbf{z}}$ and find the best $\mathbf{W}$ that approximates $\mathbf{W}\hat{\mathbf{x}}_i \approx \mathbf{x}_i$ in a least-squares sense. In the proportional regime $\frac{n}{d} \rightarrow κ> 1$ we use the {\it Convex Gaussian Min-Max Theorem (CGMT)} to analytically find the closed form expression for the generalization error of the denoiser obtained from this process. Using this expression one can optimize over $\mathbfΣ_1$ to find the best possible denoiser. Our numerical simulations show that our denoiser outperforms the ``empirical" Wiener filter in many scenarios and approaches the optimal Wiener filter as $κ\rightarrow\infty$.

Precise Performance of Linear Denoisers in the Proportional Regime

Abstract

In the present paper we study the performance of linear denoisers for noisy data of the form , where is the desired data with zero mean and unknown covariance , and is additive noise. Since the covariance is not known, the standard Wiener filter cannot be employed for denoising. Instead we assume we are given samples from the true distribution. A standard approach would then be to estimate from the samples and use it to construct an ``empirical" Wiener filter. However, in this paper, motivated by the denoising step in diffusion models, we take a different approach whereby we train a linear denoiser from the data itself. In particular, we synthetically construct noisy samples of the data by injecting the samples with Gaussian noise with covariance and find the best that approximates in a least-squares sense. In the proportional regime we use the {\it Convex Gaussian Min-Max Theorem (CGMT)} to analytically find the closed form expression for the generalization error of the denoiser obtained from this process. Using this expression one can optimize over to find the best possible denoiser. Our numerical simulations show that our denoiser outperforms the ``empirical" Wiener filter in many scenarios and approaches the optimal Wiener filter as .
Paper Structure (13 sections, 5 theorems, 97 equations, 4 figures)

This paper contains 13 sections, 5 theorems, 97 equations, 4 figures.

Table of Contents

  1. Introduction and Related Works
  2. Problem Setting and Preliminaries
  3. Main Results
  4. Proof Technique and Sketches
  5. Experiments
  6. Verifying Corollary \ref{['cor: scal']} and Theorem \ref{['thm: N=1']}
  7. Heuristics for using the result of Theorem \ref{['thm: N=1']} for training denoisers
  8. Conclusion and Future Directions
  9. Proof of Theorems \ref{['thm: N=1']} and \ref{['thm: arb_N']}: common part
  10. End of the proof of Theorem \ref{['thm: N=1']}
  11. End of the Proof of Theorem \ref{['thm: arb_N']}.
  12. Proof of Concentration of the Generalization Error
  13. Proof of Corollary \ref{['cor: scal']}

Key Result

theorem 1

Assume that $N=1$ and Assumptions ass: main hold. Then, for the denoiser $\mathbf{W}_{\text{lsq}}$ found via eq: denoiser_ls, its generalization error eq: gen_err can be characterized asymptotically as follows:

Figures (4)

  • Figure 1: Phase Transition explained in Remark \ref{['rem: pht']}
  • Figure 2: Verification of Corollary \ref{['cor: scal']} with $c = 1$
  • Figure 3: Numerical verification of Theorem \ref{['thm: N=1']}
  • Figure 4: Comparison of our approach to the empirical Wiener and true Wiener filters

Theorems & Definitions (7)

  • theorem 1
  • corollary 1
  • corollary 2
  • remark 1
  • theorem 2
  • lemma 1
  • proof