UGPNet: Universal Generative Prior for Image Restoration

TL;DR

UGPNet proposes a universal generative prior framework for image restoration that unites regression-based restoration with a pretrained generative prior. It introduces three modular components—restoration, synthesis, and fusion—allowing plug-and-play use of diverse regression backbones and a StyleGAN2-based synthesis path, with a fusion stage that learns to combine structural fidelity with high-frequency textures. The method is trained in three stages and evaluated on denoising, deblurring, and super-resolution, showing strong perceptual quality (FID) while preserving structural accuracy (PSNR/SSIM) and robustness to out-of-distribution inputs. Overall, UGPNet achieves high-quality restorations with realistic textures, improved perceptual metrics, and strong robustness, highlighting the practical impact of combining regression accuracy with generative priors in a flexible framework.

Abstract

Recent image restoration methods can be broadly categorized into two classes: (1) regression methods that recover the rough structure of the original image without synthesizing high-frequency details and (2) generative methods that synthesize perceptually-realistic high-frequency details even though the resulting image deviates from the original structure of the input. While both directions have been extensively studied in isolation, merging their benefits with a single framework has been rarely studied. In this paper, we propose UGPNet, a universal image restoration framework that can effectively achieve the benefits of both approaches by simply adopting a pair of an existing regression model and a generative model. UGPNet first restores the image structure of a degraded input using a regression model and synthesizes a perceptually-realistic image with a generative model on top of the regressed output. UGPNet then combines the regressed output and the synthesized output, resulting in a final result that faithfully reconstructs the structure of the original image in addition to perceptually-realistic textures. Our extensive experiments on deblurring, denoising, and super-resolution demonstrate that UGPNet can successfully exploit both regression and generative methods for high-fidelity image restoration.

Figures (7)

• Figure 1: We present UGPNet, a universal image restoration framework that combines the benefits of an existing regression-based restoration network and a generative prior-based network. (a) Given degraded images, e.g. a blurry one, (b) a regression network NAFNet fails to recover perceptually-realistic details while it recovers the coarse structure of the original image. (c) In contrast, a generative network gfpgan synthesizes perceptually-realistic high-frequency details while sacrificing structural consistency with the input image. (d) UGPNet allows us to maintain the original structure of the input image and synthesize perceptually-realistic high-frequency details. As a universal framework, (f) UGPNet is applicable to natural images lsun. In addition, (g) it is robust against catastrophic failures that generative prior-based methods encounter when restoring images outside the training distributions.
• Figure 2: UGPNet consists of three sub-modules: restoration, synthesis, and fusion modules. Given a degraded input image $x$, the restoration module first recovers the original image structure exploiting a regression network. On top of the regressed output, the synthesis module synthesizes high-frequency details exploiting a generative network. Lastly, the fusion module combines the latent features from both modules to generate a final restored image $\hat{x}$.
• Figure 3: An example of image outputs of UGPNet's modules and corresponding ground-truth image. On top of the image structure of restoration module output (a) $x_{reg}$, the fusion module brings high-frequency details of synthesis module output (b) $x_{syn}$ to generate the final output (c) $\hat{x}$.
• Figure 4: UGPNet allows flexible selection of diverse regression networks in the restoration module. We show restoration results using regression models (UNet UNet, HINet hinet, NAFNet NAFNet and RRDBNet esrgan) on the top row for (a) deblurring and (b) super-resolution. We can equip any regression models into UGPNet that synthesizes perceptually-realistic high-frequency details, as shown in the bottom row.
• Figure 5: Qualitative comparison of (a) denoising and (b) deblurring methods: regression methods (Uformer uformer_multiple and NAFNet NAFNet), generative methods (GFP-GAN gfpgan, GPEN gpen, VQFR VQFR), and UGPNet with NAFNet NAFNet. UGPNet recovers authentic image structure and colors compared to generative methods while synthesizing sharp high-frequency details compared to regression methods.