Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Image-Adaptive GAN based Reconstruction

Shady Abu Hussein, Tom Tirer, Raja Giryes

Abstract

In the recent years, there has been a significant improvement in the quality of samples produced by (deep) generative models such as variational auto-encoders and generative adversarial networks. However, the representation capabilities of these methods still do not capture the full distribution for complex classes of images, such as human faces. This deficiency has been clearly observed in previous works that use pre-trained generative models to solve imaging inverse problems. In this paper, we suggest to mitigate the limited representation capabilities of generators by making them image-adaptive and enforcing compliance of the restoration with the observations via back-projections. We empirically demonstrate the advantages of our proposed approach for image super-resolution and compressed sensing.

Image-Adaptive GAN based Reconstruction

Abstract

In the recent years, there has been a significant improvement in the quality of samples produced by (deep) generative models such as variational auto-encoders and generative adversarial networks. However, the representation capabilities of these methods still do not capture the full distribution for complex classes of images, such as human faces. This deficiency has been clearly observed in previous works that use pre-trained generative models to solve imaging inverse problems. In this paper, we suggest to mitigate the limited representation capabilities of generators by making them image-adaptive and enforcing compliance of the restoration with the observations via back-projections. We empirically demonstrate the advantages of our proposed approach for image super-resolution and compressed sensing.

Paper Structure

This paper contains 11 sections, 2 theorems, 11 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Proposed Method
  4. An Image-Adaptive Approach
  5. Mathematical Motivation for Image-Adaptation
  6. "Hard" vs. "Soft" Compliance to Observations
  7. Experiments
  8. Compressed Sensing
  9. Super-Resolution
  10. Deblurring
  11. Conclusion

Key Result

Theorem 1

Let $\mathbf{G}(\mathbf{z}):\mathbb{R}^k \rightarrow \mathbb{R}^n$ as given in Eq_generator, $\mathbf{A} \in \mathbb{R}^{m \times n}$ with $A_{ij}\sim \mathcal{N}(0,1/m)$, $m=\Omega \left ( kL\mathrm{log}n \right )$, and $\mathbf{y}=\mathbf{A}\mathbf{x}+\mathbf{e}$. Let $\hat{\mathbf{z}}$ minimize $ where $E_{rep}(\mathbf{G}(\cdot),\mathbf{x}) := \underset{\mathbf{z}}{\textrm{min}}\|\mathbf{G}(\ma

Figures (8)

  • Figure 1: Compressed sensing with Gaussian measurement matrix using BEGAN. From left to right and top to bottom: original image, CSGM for $m/n=0.122$, CSGM-BP for $m/n=0.122$, CSGM for $m/n=0.61$, CSGM-BP for $m/n=0.61$, IAGAN for $m/n=0.122$, IAGAN-BP for $m/n=0.122$, IAGAN for $m/n=0.61$, IAGAN-BP for $m/n=0.61$.
  • Figure 2: Compressed sensing with Gaussian measurement matrix using BEGAN. Reconstruction MSE (averaged over 100 images from CelebA) vs. the compression ratio $m/n$.
  • Figure 3: Compressed sensing with 30% subsampled Fourier measurements and noise level of 10/255, for CelebA-HQ images. From left to right and top to bottom: original image, naive reconstruction (zero padding and IFFT), DIP, CSGM, and IAGAN. Note that CSGM and IAGAN use the PGGAN prior.
  • Figure 4: Binary masks for compressed sensing with 30% (left) and 50% (right) subsampled Fourier measurements.
  • Figure 5: Super-resolution using PGGAN with scale factor of 16 without noise. From left to right and top to bottom: original image, bicubic upsampling, DIP, CSGM, IAGAN, CSGM-BP, and IAGAN-BP.
  • ...and 3 more figures

Theorems & Definitions (3)

  • Theorem 1
  • Proposition 2
  • proof