Why the noise model matters: A performance gap in learned regularization

TL;DR

The paper investigates learning regularizers for linear inverse problems by comparing learned variational methods to the optimal affine reconstruction (LMMSE) under unknown noise statistics. It shows that when noise covariance is learned, the regularization problem can recover the LMMSE estimator, making several regularizers (Lavrentiev, Quadratic, Tikhonov with weights) equivalent in performance to the affine optimum; without learning the noise model, there is a tangible performance gap, and Lavrentiev regularizers can be non-symmetric while quadratic ones may be indefinite. The authors derive explicit optimal forms for Lavrentiev ($M = A^{T}\Sigma_{\varepsilon}A(A^{T}A)^{-1}\Sigma_{x^{\dag}}^{-1}$) and quadratic ($M=N^{-1}-A^{T}A$, with $N$ solving a Lyapunov equation) regularizers, and they provide theoretical conditions under which these approaches match LMMSE; numerical experiments on deconvolution and speech dereverberation corroborate the existence of gaps when noise is non-white and demonstrate the value of co-learning noise statistics with regularizers. Collectively, the work highlights the practical importance of accurate noise modeling or co-learning in data-driven regularization and clarifies the limits of standard regularizers when the noise model is not known.

Abstract

This article addresses the challenge of learning effective regularizers for linear inverse problems. We analyze and compare several types of learned variational regularization against the theoretical benchmark of the optimal affine reconstruction, i.e. the best possible affine linear map for minimizing the mean squared error. It is known that this optimal reconstruction can be achieved using Tikhonov regularization, but this requires precise knowledge of the noise covariance to properly weight the data fidelity term. However, in many practical applications, noise statistics are unknown. We therefore investigate the performance of regularization methods learned without access to this noise information, focusing on Tikhonov, Lavrentiev, and quadratic regularization. Our theoretical analysis and numerical experiments demonstrate that for non-white noise, a performance gap emerges between these methods and the optimal affine reconstruction. Furthermore, we show that these different types of regularization yield distinct results, highlighting that the choice of regularizer structure is critical when the noise model is not explicitly learned. Our findings underscore the significant value of accurately modeling or co-learning noise statistics in data-driven regularization.

Figures (5)

• Figure 1: Left: The empirical mean of the data of experiment 1. Right: One sample of the data $x^\dag$ and the corresponding $y = Ax^{\dag} + \varepsilon$.
• Figure 2: The empirical covariance matrix $\Sigma_{x^{\dag}}$ of the data $x^{\dag}$.
• Figure 3: First row: Plot of the optimal matrix $M$ for Lavrentiev regularization. Bottom row: Plot of the optimal matrix $M$ for quadratic regularization. The right column shows a zoom on the top left $40\times 40$ block of the matrices on the left.
• Figure 4: Caption
• Figure 5: Illustration of the learning progress using a warm-start strategy in the transition from method to method. The different colors indicate different noise levels, as shown on the left vertical axis. While optimization variables are carried over during each warm-start, the learning rate decay is the same for all four methods, as indicated by the dotted line. All error curves are smoothed using a moving average of length 6610, corresponding to 5% of the length of one epoch.

Theorems & Definitions (16)

• Lemma 3.1
• proof
• Remark 3.2
• Corollary 3.3: Optimal affine offset
• Corollary 3.4: LMMSE estimation
• proof
• Theorem 4.1: Optimal Tikhonov regularization and noise weight
• proof
• Theorem 5.1: Optimal Lavrentiev regularization
• proof