Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation

TL;DR

This work introduces Deep Generative Prior (DGP), a universal image prior derived from a pre-trained GAN on large natural-image datasets, to enable versatile restoration and manipulation. By relaxing GAN inversion to allow on-the-fly fine-tuning of both the generator and latent vector, and by enforcing discriminator-guided feature matching with a progressive, block-wise optimization, DGP preserves natural image statistics while recovering missing semantics. The approach achieves high-quality restoration (colorization, inpainting, super-resolution) and enables compelling manipulation (random jittering, morphing, category transfer) on complex datasets like ImageNet, outperforming traditional GAN-inversion baselines. The paper also provides detailed implementation guidance for applying DGP in practical settings and demonstrates its broader applicability beyond single-image priors.

Abstract

Learning a good image prior is a long-term goal for image restoration and manipulation. While existing methods like deep image prior (DIP) capture low-level image statistics, there are still gaps toward an image prior that captures rich image semantics including color, spatial coherence, textures, and high-level concepts. This work presents an effective way to exploit the image prior captured by a generative adversarial network (GAN) trained on large-scale natural images. As shown in Fig.1, the deep generative prior (DGP) provides compelling results to restore missing semantics, e.g., color, patch, resolution, of various degraded images. It also enables diverse image manipulation including random jittering, image morphing, and category transfer. Such highly flexible restoration and manipulation are made possible through relaxing the assumption of existing GAN-inversion methods, which tend to fix the generator. Notably, we allow the generator to be fine-tuned on-the-fly in a progressive manner regularized by feature distance obtained by the discriminator in GAN. We show that these easy-to-implement and practical changes help preserve the reconstruction to remain in the manifold of nature image, and thus lead to more precise and faithful reconstruction for real images. Code is available at https://github.com/XingangPan/deep-generative-prior.

Figures (24)

• Figure 1: These image restoration(a)(b)(c)(d) and manipulation(e)(f)(g) effects are achieved by leveraging the rich generative prior of a GAN. The GAN does not see these images during training
• Figure 2: Comparison of various methods in reconstructing a gray image under the gray-scale observation space using a GAN. Conventional GAN-inversion strategies like (b)zhu2016generative and (c)bau2019seeing produce imprecise reconstruction for the existing semantics. In this work, we relax the generator so that it can be fine-tuned on-the-fly, achieving more accurate reconstruction as in (d)(e)(f), of which optimization is based on (d) VGG perceptual loss, (e) discriminator feature matching loss, and (f) combined with progressive reconstruction, respectively. We highlight that discriminator is important to preserve the generative prior so as to achieve better restoration for the missing information (i.e., color). The proposed progressive strategy eliminates the 'information lingering' artifacts as in the red box in (e)
• Figure 3: Comparison of different loss types when fine-tuning the generator to reconstruct the image
• Figure 4: Progressive reconstruction of the generator can better preserves the consistency between missing and existing semantics in comparison to simultaneous fine-tuning on all the parameters at once. Here the list of images shown in the middle are the outputs of the generator in different fine-tuning stages.
• Figure 5: Colorization. Qualitative comparison of Autocolorize larsson2016learning, other GAN-inversion methods bau2019seeingbau2019semantic, and our DGP