Convergence rates for Tikhonov regularization on compact sets: application to neural networks

TL;DR

The paper introduces a regularization framework for ill-posed inverse problems by minimizing the Tikhonov functional on a dense sequence of compact sets within a weakly closed constraint, treating the set index as an additional regularization parameter. It proves stability and convergence with rates matching classical Tikhonov theory, achieving optimal rates in the linear case, and extends the method to neural-network parametrizations by restricting to bounded-weight networks that approximate the true solution while enforcing nonnegativity. A density result shows that neural networks with bounded weights densely populate the positive $L^2$ space, enabling a practical NN-based regularization scheme; a detailed rate analysis yields an $\mathcal{O}(\delta^{2/3})$ convergence under suitable conditions. Numerical experiments on Computerized Tomography with the Radon transform demonstrate edge-preserving reconstructions and superior performance of the NN-regularized approach over classical Tikhonov at comparable noise levels, validating the theoretical findings and illustrating potential for nonconvex optimization advantages in inverse problems.

Abstract

In this work, we consider ill-posed inverse problems in which the forward operator is continuous and weakly closed, and the sought solution belongs to a weakly closed constraint set. We propose a regularization method based on minimizing the Tikhonov functional on a sequence of compact sets which is dense in the intersection between the domain of the forward operator and the constraint set. The index of the compact sets can be interpreted as an additional regularization parameter. We prove that the proposed method is a regularization, achieving the same convergence rates as classical Tikhonov regularization and attaining the optimal convergence rate when the forward operator is linear. Moreover, we show that our methodology applies to the case where the constrained solution space is parametrized by means of neural networks (NNs), and the constraint is obtained by composing the last layer of the NN with a suitable activation function. In this case the dense compact sets are defined by taking a family of bounded weight NNs with increasing weight bound. Finally, we present some numerical experiments in the case of Computerized Tomography to compare the theoretical behavior of the reconstruction error with that obtained in a finite dimensional and non-asymptotic setting. The numerical tests also show that our NN-based regularization method is able to provide piece-wise constant solutions and to preserve the sharpness of edges, thus achieving lower reconstruction errors compared to the classical Tikhonov approach for the same level of noise in the data.

Figures (3)

• Figure 1: Shepp-Logan phantom and corresponding sinogram (left and right panel, respectively).
• Figure 2: Comparison of the NN reconstrction and the Tikhonov reconsturction for the first, 5 and the last value of noise. Low noise level, medium noise level, and high noise level correspond to SNR values of 42.60, 23.10, and 16.58, respectively.
• Figure 3: Average reconstruction error over $5$ different noise realization for both our proposed method and the Tikhonov method, as a function of the noise level $\delta$ (left and right panel, respectively). The error bars represent the standard deviation across the same noise realizations. For comparison with optimal convergence rates, a quantity proportional to $\delta^{2/3}$ is plotted with a dashed line in both panels. The values $c_1$ and $c_2$ are determined by means the least square method.

Theorems & Definitions (22)

• Definition 2.1
• Proposition 2.1
• Proof 1
• Definition 2.2
• Theorem 2.1
• Proof 2
• Theorem 2.2
• Proof 3
• Theorem 2.3
• Proof 4