Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Small noise analysis for Tikhonov and RKHS regularizations

Quanjun Lang, Fei Lu

TL;DR

A small noise analysis framework is established to assess the effects of norms in Tikhonov and RKHS regularizations, in the context of ill-posed linear inverse problems with Gaussian noise, and proposes an innovative class of adaptive fractional RK HS regularizers.

Abstract

Regularization plays a pivotal role in ill-posed machine learning and inverse problems. However, the fundamental comparative analysis of various regularization norms remains open. We establish a small noise analysis framework to assess the effects of norms in Tikhonov and RKHS regularizations, in the context of ill-posed linear inverse problems with Gaussian noise. This framework studies the convergence rates of regularized estimators in the small noise limit and reveals the potential instability of the conventional L2-regularizer. We solve such instability by proposing an innovative class of adaptive fractional RKHS regularizers, which covers the L2 Tikhonov and RKHS regularizations by adjusting the fractional smoothness parameter. A surprising insight is that over-smoothing via these fractional RKHSs consistently yields optimal convergence rates, but the optimal hyper-parameter may decay too fast to be selected in practice.

Small noise analysis for Tikhonov and RKHS regularizations

TL;DR

A small noise analysis framework is established to assess the effects of norms in Tikhonov and RKHS regularizations, in the context of ill-posed linear inverse problems with Gaussian noise, and proposes an innovative class of adaptive fractional RK HS regularizers.

Abstract

Regularization plays a pivotal role in ill-posed machine learning and inverse problems. However, the fundamental comparative analysis of various regularization norms remains open. We establish a small noise analysis framework to assess the effects of norms in Tikhonov and RKHS regularizations, in the context of ill-posed linear inverse problems with Gaussian noise. This framework studies the convergence rates of regularized estimators in the small noise limit and reveals the potential instability of the conventional L2-regularizer. We solve such instability by proposing an innovative class of adaptive fractional RKHS regularizers, which covers the L2 Tikhonov and RKHS regularizations by adjusting the fractional smoothness parameter. A surprising insight is that over-smoothing via these fractional RKHSs consistently yields optimal convergence rates, but the optimal hyper-parameter may decay too fast to be selected in practice.
Paper Structure (17 sections, 8 theorems, 61 equations, 2 figures)

This paper contains 17 sections, 8 theorems, 61 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Notations.
  3. Related works.
  4. Tikhonov and RKHS regularization
  5. Ill-posed linear inverse problems
  6. Examples
  7. Adaptive fractional Sobolev spaces and regularization
  8. Small noise analysis for regularized estimators
  9. Finite-rank operators: removing bias outside the FSOI
  10. Infinite-rank operator: convergence rate
  11. Numerical Examples
  12. Oracle rates under exponential spectral decay
  13. Rates in practice under polynomial spectral decay.
  14. Limitations
  15. Conclusion
  16. ...and 2 more sections

Key Result

Lemma 2.4

Under Assumption assumption:LG, the mini-norm LSE is the unique minimizer of the loss function in eq:lossFn in the space $H$, and it can be written as Let $\phi_*$ be the true solution, i.e., $y_k = L_k \phi_*+\sigma \dot W_k$. The following statements hold true.

Figures (2)

  • Figure 1: over-smoothing ($s>r - \frac{\beta+1}{2}$) yields the optimal convergence rate (left), at the price of fast decaying optimal hyper-parameter $\lambda_*$ (right). Here $s$ is the smoothness parameter of the fractional Sobolev space for regularization, $r$ is the regularity of the true function, and $\beta$ represents the spectral decay of the normal operator. See Sect.\ref{['sec:oracle-rate']} and Theorem \ref{['thm:snl_reguEst']} for details.
  • Figure 2: over-smoothing ($s=2$) makes it difficult to select the optimal $\lambda_*$ by the L-curve method (right), leading to a relatively large error (left). However, when properly regularized with $s=1$, the L-curve $\lambda_*$'s are close to the oracle ones and lead to estimators that are significantly more accurate than those with $s=0$ or $s=2$.

Theorems & Definitions (12)

  • Definition 2.2: Function space of identifiability (FSOI)
  • Definition 2.3: Ill-posed inverse problem
  • Lemma 2.4: The mini-norm LSE
  • Definition 2.5: Adaptive fractional Sobolev spaces
  • Proposition 2.6: Fractional Sobolev space regularized estimator
  • Proposition 3.1
  • Theorem 3.3: Convergence rates of regularized estimators
  • Lemma 3.4
  • Lemma 3.5
  • Remark 3.6
  • ...and 2 more