Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Inverse problem regularization with hierarchical variational autoencoders

Jean Prost, Antoine Houdard, Andrés Almansa, Nicolas Papadakis

TL;DR

This work addresses ill-posed inverse problems by regularizing with a hierarchical variational autoencoder (HVAE) prior. It introduces PnP-HVAE, an encoder-guided, joint-posterior optimization that avoids backpropagation through the generator and leverages temperature scaling to control regularization strength across HVAE levels. The method provides convergence guarantees under contractivity assumptions and demonstrates strong restoration performance on both face datasets (via a pre-trained VDVAE) and natural images using a patch-based HVAE (PatchVDVAE). Overall, PnP-HVAE achieves competitive results with state-of-the-art denoiser-based and generative-model-based methods while enabling restoration across image sizes and maintaining fidelity to observations.

Abstract

In this paper, we propose to regularize ill-posed inverse problems using a deep hierarchical variational autoencoder (HVAE) as an image prior. The proposed method synthesizes the advantages of i) denoiser-based Plug \& Play approaches and ii) generative model based approaches to inverse problems. First, we exploit VAE properties to design an efficient algorithm that benefits from convergence guarantees of Plug-and-Play (PnP) methods. Second, our approach is not restricted to specialized datasets and the proposed PnP-HVAE model is able to solve image restoration problems on natural images of any size. Our experiments show that the proposed PnP-HVAE method is competitive with both SOTA denoiser-based PnP approaches, and other SOTA restoration methods based on generative models.

Inverse problem regularization with hierarchical variational autoencoders

TL;DR

This work addresses ill-posed inverse problems by regularizing with a hierarchical variational autoencoder (HVAE) prior. It introduces PnP-HVAE, an encoder-guided, joint-posterior optimization that avoids backpropagation through the generator and leverages temperature scaling to control regularization strength across HVAE levels. The method provides convergence guarantees under contractivity assumptions and demonstrates strong restoration performance on both face datasets (via a pre-trained VDVAE) and natural images using a patch-based HVAE (PatchVDVAE). Overall, PnP-HVAE achieves competitive results with state-of-the-art denoiser-based and generative-model-based methods while enabling restoration across image sizes and maintaining fidelity to observations.

Abstract

In this paper, we propose to regularize ill-posed inverse problems using a deep hierarchical variational autoencoder (HVAE) as an image prior. The proposed method synthesizes the advantages of i) denoiser-based Plug \& Play approaches and ii) generative model based approaches to inverse problems. First, we exploit VAE properties to design an efficient algorithm that benefits from convergence guarantees of Plug-and-Play (PnP) methods. Second, our approach is not restricted to specialized datasets and the proposed PnP-HVAE model is able to solve image restoration problems on natural images of any size. Our experiments show that the proposed PnP-HVAE method is competitive with both SOTA denoiser-based PnP approaches, and other SOTA restoration methods based on generative models.
Paper Structure (42 sections, 9 theorems, 67 equations, 17 figures, 4 tables, 2 algorithms)

This paper contains 42 sections, 9 theorems, 67 equations, 17 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions and outline
  3. Related works
  4. Plug-and-Play methods
  5. Deep generative models for inverse problems
  6. Background on variational autoencoders
  7. VAE training
  8. Hierarchical variational autoencoders
  9. Temperature scaling
  10. Regularization with HVAE Prior
  11. Tempered hierarchical joint posterior
  12. Encoder approximation of the joint posterior
  13. Alternate optimization with PnP-HVAE
  14. Convergence analysis
  15. Plug-and-Play HVAE
  16. ...and 27 more sections

Key Result

Proposition 1

Assume the decoder is defined as in eq:decoder_constant. Denote $\text{HVAE}(\bm{x}, \bm{\tau}) := \mu_{\theta}\left(E_{\bm{\tau}}\left(\bm{x}\right)\right)$. Then the alternate scheme described in Algorithm algo:final writes

Figures (17)

  • Figure 1: Numerical estimation of the Lipschitz constant of PatchVDVAE reconstruction with different temperatures $\tau$. We present the histogram of ratio values $\frac{||\text{HVAE}(\bf{u}, \tau)-\text{HVAE}(\bf{v}, \tau)||}{||\bf{u}-\bf{v}||}$, where $\bf{u}$ and $\bf{v}$ are natural images corrupted with white Gaussian noise of different standard deviations $\sigma$. For noisy images ($\sigma>0)$, the observed Lipschitz constant is always less than $1$.
  • Figure 2: Comparison of the convergence of PnP-HVAE algorithm \ref{['algo:final']} with respect to the baseline Adam optimizer, on a deblurring problem. Left (Convergence of the function value): PnP-HVAE converges faster to a minimum of the joint posterior $J_1(\bm{x}_k,\bm{z}_k)$ in \ref{['eq:J1']}. Right (Convergence of iterates $\bm{x}_k$): PnP-HVAE is more stable than Adam.
  • Figure 3: Visual comparaison of image restoration methods based on deep generative models. We studied $3$ tasks on face images: inpainting (top), deblurring (middle), super-resolution (bottom). Contrary to the optimization of the objective \ref{['eq:J1']} with Adam, our alternate algorithm generates realistic results, on par with ILO daras2021intermediate, while remaining consistant with the observation.
  • Figure 4: Left: $64\times64$ patches samples from our patchVDVAE model trained on patches from natural images. Right: PatchVDVAE is fully convolutional and it can generate images of higher resolution (here: $128\times128$).
  • Figure 5: Natural image deblurring
  • ...and 12 more figures

Theorems & Definitions (9)

  • Proposition 1
  • Proposition 2: Proof in supplementary
  • Corollary 1
  • Proposition 3: Proof in supplementary
  • Proposition 4
  • Proposition 5: Algorithm \ref{['algo:hierarchical_latent_reg']} computes the global minimum of $J_2(\bm{x},\bm{z})$ with respect to $\bm{z}$
  • Proposition 6: Tweedie's formula for HVAEs.
  • Proposition 7
  • Corollary 2