Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Convergence of A Data-Driven Regularized Stochastic Gradient Descent for Nonlinear Ill-Posed Problems

Zehui Zhou

TL;DR

This work analyze a new data-driven regularized stochastic gradient descent for the efficient numerical solution of a class of nonlinear ill-posed inverse problems in infinite dimensional Hilbert spaces and proves the regularizing property of this method under the tangential cone condition and a priori parameter choice.

Abstract

Stochastic gradient descent (SGD) is a promising method for solving large-scale inverse problems, due to its excellent scalability with respect to data size. In this work, we analyze a new data-driven regularized stochastic gradient descent for the efficient numerical solution of a class of nonlinear ill-posed inverse problems in infinite dimensional Hilbert spaces. At each step of the iteration, the method randomly selects one equation from the nonlinear system combined with a corresponding equation from the learned system based on training data to obtain a stochastic estimate of the gradient and then performs a descent step with the estimated gradient. We prove the regularizing property of this method under the tangential cone condition and a priori parameter choice and then derive the convergence rates under the additional source condition and range invariance conditions. Several numerical experiments are provided to complement the analysis.

On the Convergence of A Data-Driven Regularized Stochastic Gradient Descent for Nonlinear Ill-Posed Problems

TL;DR

This work analyze a new data-driven regularized stochastic gradient descent for the efficient numerical solution of a class of nonlinear ill-posed inverse problems in infinite dimensional Hilbert spaces and proves the regularizing property of this method under the tangential cone condition and a priori parameter choice.

Abstract

Stochastic gradient descent (SGD) is a promising method for solving large-scale inverse problems, due to its excellent scalability with respect to data size. In this work, we analyze a new data-driven regularized stochastic gradient descent for the efficient numerical solution of a class of nonlinear ill-posed inverse problems in infinite dimensional Hilbert spaces. At each step of the iteration, the method randomly selects one equation from the nonlinear system combined with a corresponding equation from the learned system based on training data to obtain a stochastic estimate of the gradient and then performs a descent step with the estimated gradient. We prove the regularizing property of this method under the tangential cone condition and a priori parameter choice and then derive the convergence rates under the additional source condition and range invariance conditions. Several numerical experiments are provided to complement the analysis.
Paper Structure (26 sections, 24 theorems, 216 equations, 3 figures, 9 tables, 1 algorithm)

This paper contains 26 sections, 24 theorems, 216 equations, 3 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Main results and discussions
  3. Convergence of the data-driven SGD
  4. Convergence for exact data
  5. Convergence for noisy data
  6. Convergence rates
  7. Recursion on the mean
  8. Stochastic error
  9. Convergence rates
  10. Numerical experiments
  11. Linear inverse problems
  12. Order optimality of data-driven SGD
  13. Dependence on the regularization parameter
  14. Dependence on the data-driven model
  15. Nonlinear inverse problems
  16. ...and 11 more sections

Key Result

Theorem 2.1

Let Assumptions ass:sol(i)-(iii) and ass:stepsize(i) be fulfilled with $L_F^2\eta_k<1-\eta_F$ for any $k\geq1$. If the condition $\lim_{\delta\to 0^+}\lambda_k^\delta=\lambda_k^0$ holds for any $k\in\mathbb{N}$ and the stopping index $k(\delta)\in\mathbb{N}$ is chosen such that then for the data-driven SGD iterate $x_k^\delta$ in eqn:datasgd, there exists a solution $x^*\in\mathcal{B}_\rho(x^\dag

Figures (3)

  • Figure 1: Singular Value Spectrum
  • Figure 2: The convergence of relative mean squared errors $e=\frac{\mathbb{E}[\|x_k^\delta-x^\dag\|^2]}{\|x^\dag\|^2}$ of four methods for squared-phillips.
  • Figure 3: The convergence of relative mean squared errors $e=\frac{\mathbb{E}[\|x_k^\delta-x^\dag\|^2]}{\|x^\dag\|^2}$ of four methods for squared-shaw.

Theorems & Definitions (55)

  • Theorem 2.1: Convergence for noisy data
  • Theorem 2.2
  • Proposition 3.1
  • Corollary 3.1
  • proof
  • Proposition 3.2
  • Theorem 3.1: Convergence for exact data
  • proof
  • Corollary 3.2
  • proof
  • ...and 45 more