Reducing the Representation Error of GAN Image Priors Using the Deep Decoder

TL;DR

This work tackles the representation error of GAN priors in imaging inverse problems by proposing a lightweight hybrid prior that linearly combines a fixed GAN with an unlearned Deep Decoder: $H(z, \theta, \alpha, \beta)= \alpha G_\phi(z) + \beta DD(\theta)$. By optimizing over $z$, $\theta$, $\alpha$, and $\beta$, the method achieves higher PSNR than either component alone in compressive sensing and image super-resolution, especially in-distribution, while remaining underparameterized and computationally cheap. Compared to overparametrized alternatives like GAN-as-DIP or IAGAN, the hybrid offers competitive or superior performance with far fewer parameters, and adapts gracefully to out-of-distribution data by reducing the GAN contribution when needed. The approach provides a practical, extensible framework for leveraging both learned and unlearned priors in inverse problems, with strong implications for robust, high-quality image recovery under varying measurements and distributions.

Abstract

Generative models, such as GANs, learn an explicit low-dimensional representation of a particular class of images, and so they may be used as natural image priors for solving inverse problems such as image restoration and compressive sensing. GAN priors have demonstrated impressive performance on these tasks, but they can exhibit substantial representation error for both in-distribution and out-of-distribution images, because of the mismatch between the learned, approximate image distribution and the data generating distribution. In this paper, we demonstrate a method for reducing the representation error of GAN priors by modeling images as the linear combination of a GAN prior with a Deep Decoder. The deep decoder is an underparameterized and most importantly unlearned natural signal model similar to the Deep Image Prior. No knowledge of the specific inverse problem is needed in the training of the GAN underlying our method. For compressive sensing and image superresolution, our hybrid model exhibits consistently higher PSNRs than both the GAN priors and Deep Decoder separately, both on in-distribution and out-of-distribution images. This model provides a method for extensibly and cheaply leveraging both the benefits of learned and unlearned image recovery priors in inverse problems.

Figures (11)

• Figure 1: Our hybrid model, a combination of a GAN prior and a Deep Decoder, has significantly less representation error than the GAN Prior alone.
• Figure 2: The model includes parameters $\alpha$, $\beta$, $\theta$, and $z$, which together comprise the image representation enforced by our Hybrid model. The final output information is a learned linear combination of the two component images.
• Figure 3: Reconstruction PSNRs versus measurement numbers for in-distribution test images for our hybrid model, a Deep Decoder, and a GAN prior. The left panels zoom in to the low measurement regime of the right panels. Our hybrid model is able to yield higher PSNRs than both of its components, the Deep Decoder and a GAN, on in-distribution test images, in all but the lowest measurement regime. The effect is replicated both for the BEGAN (top row) and the DCGAN (bottom row).
• Figure 4: Left: Samples of reconstructed images for $m=2500$ measurements, a compression ratio of $0.051$. The GAN prior has significant representation error, to the point where it appears to recover the face of the wrong person. The Deep Decoder has artifacts arising from too much smoothing of facial features. The hybrid model has sharply defined features, as does the GAN, without the unnecessary smoothness of the Deep Decoder by itself. Right: A comparison of output examples for the Deep Decoder and hybrid models, along with the two components underlying the hybrid model. The difference is detail is noticeable between the pure Deep Decoder and the Hybrid model.
• Figure 5: Left: Performance of various image models on compressed sensing of images of birds, using a GAN prior trained to generate images of celebrity faces. Surprisingly, the hybrid model still marginally outperforms the Deep Decoder, as the GAN prior learns some general image statistics which are beneficial. Right: Coefficients of the GAN and Deep Decoder in the hybrid $H$. Darker colors correspond to images out of the GAN prior training distribution. As one would expect, the coefficient of the GAN prior is diminished when reconstructing images for which the GAN has not learned relevant features.