Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Over Complete Deep Learning Method for Inverse Problems

Moshe Eliasof, Eldad Haber, Eran Treister

TL;DR

This work addresses ill-posed inverse problems by embedding the unknown solution into a higher-dimensional space via a learnable embedding $x=E z$ and learning a regularizer $\phi(z,\theta)$ on the embedded variable. It derives two neural architectures, OPTEnet and EUnet, that perform gradient-like updates in the embedded space and can be unrolled into time-evolving networks, providing a data-driven, end-to-end regularization strategy. Empirical results across duathlon, image deblurring, and magnetics demonstrate that embedding-based methods outperform end-to-end proximal/regularization approaches and diffusion models, especially for highly ill-posed problems where the original landscape is highly nonconvex. The paper highlights the potential of combining classical over-complete dictionary ideas with modern deep learning to obtain friendlier optimization surfaces and faster convergence, suggesting avenues for integrating learned embeddings with diffusion priors in future work.

Abstract

Obtaining meaningful solutions for inverse problems has been a major challenge with many applications in science and engineering. Recent machine learning techniques based on proximal and diffusion-based methods have shown promising results. However, as we show in this work, they can also face challenges when applied to some exemplary problems. We show that similar to previous works on over-complete dictionaries, it is possible to overcome these shortcomings by embedding the solution into higher dimensions. The novelty of the work proposed is that we jointly design and learn the embedding and the regularizer for the embedding vector. We demonstrate the merit of this approach on several exemplary and common inverse problems.

An Over Complete Deep Learning Method for Inverse Problems

TL;DR

This work addresses ill-posed inverse problems by embedding the unknown solution into a higher-dimensional space via a learnable embedding and learning a regularizer on the embedded variable. It derives two neural architectures, OPTEnet and EUnet, that perform gradient-like updates in the embedded space and can be unrolled into time-evolving networks, providing a data-driven, end-to-end regularization strategy. Empirical results across duathlon, image deblurring, and magnetics demonstrate that embedding-based methods outperform end-to-end proximal/regularization approaches and diffusion models, especially for highly ill-posed problems where the original landscape is highly nonconvex. The paper highlights the potential of combining classical over-complete dictionary ideas with modern deep learning to obtain friendlier optimization surfaces and faster convergence, suggesting avenues for integrating learned embeddings with diffusion priors in future work.

Abstract

Obtaining meaningful solutions for inverse problems has been a major challenge with many applications in science and engineering. Recent machine learning techniques based on proximal and diffusion-based methods have shown promising results. However, as we show in this work, they can also face challenges when applied to some exemplary problems. We show that similar to previous works on over-complete dictionaries, it is possible to overcome these shortcomings by embedding the solution into higher dimensions. The novelty of the work proposed is that we jointly design and learn the embedding and the regularizer for the embedding vector. We demonstrate the merit of this approach on several exemplary and common inverse problems.
Paper Structure (15 sections, 2 theorems, 35 equations, 12 figures, 2 tables)

This paper contains 15 sections, 2 theorems, 35 equations, 12 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Mathematical Background and Motivation
  3. Reformulating the Solution of Inverse Problems by Embedding and deep learning
  4. High Dimensional Solution Embedding
  5. Learnable Embedding and Regularization in high dimensions
  6. From Optimization to Network Architectures
  7. Network architectures and training
  8. Architectures
  9. Training
  10. Numerical Experiments
  11. The dualthlon problem
  12. Image deblurring
  13. Magnetics
  14. Summary and conclusions
  15. Additional visualizations and convergence plot

Key Result

Theorem 3.1

Let $f({\bf x})$ be a bounded function from $\mathbb{R}^N$ to $\mathbb{R}$ with continuous second derivatives. Let ${\bf x}_1$ and ${\bf x}_2$ be local minima of the function, that is Then, there exists a path (a mountain pass) defined by such that the function on this path $f({\bf x}(t))$ have a unique maximum.

Figures (12)

  • Figure 1: The duathlon problem, estimate the bike and run time from the total competition time.
  • Figure 2: Experiments with the dualthlon problem. The prior is made of three Gaussians and the data (yellow point) are presented on the left along with all possible points that fit the data (red line). Recovering a model given the data by sampling the posterior using diffusion is presented on the right (magenta points).
  • Figure 3: A non-convex prior with 2 local minima in $x$ is replaced with a learned quasi-convex prior in higher dimensions. Plots (c) and (d) are in log scale.
  • Figure 4: The potential function $\phi$ with a different number of layers.
  • Figure 5: An example of the recovery of deblurred images from the STL-10 data set. (a) Ground truth (same for all rows) (b) Observed data, (c) Diffusion, (d) Proximal, (e) EUnet. Table \ref{['tab2']} reports numerical recovery results. Additional examples are provided in Appendix \ref{['appendix:examples']}.
  • ...and 7 more figures

Theorems & Definitions (5)

  • Example 2.1
  • Theorem 3.1: Mountain pass
  • Theorem 3.2: The Mountain Bypass
  • proof
  • Example 3.3