Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization

TL;DR

PASD tackles Real-ISR and personalized stylization by injecting pixel-level conditioning into a pre-trained latent diffusion model. It combines a Degradation Removal module, Pixel-Aware Cross Attention (PACA), an Adjustable Noise Schedule (ANS), and high-level semantic information to preserve structures while generating realistic textures. The method avoids image-space training, supports switching to personalized stylizations, and demonstrates superior perceptual metrics and user-preference in Real-ISR, cartoonization, and old-photo restoration tasks. Ablation studies confirm the importance of each component, and results suggest PASD offers a flexible, scalable approach for pixel-accurate diffusion-based restoration.

Abstract

Diffusion models have demonstrated impressive performance in various image generation, editing, enhancement and translation tasks. In particular, the pre-trained text-to-image stable diffusion models provide a potential solution to the challenging realistic image super-resolution (Real-ISR) and image stylization problems with their strong generative priors. However, the existing methods along this line often fail to keep faithful pixel-wise image structures. If extra skip connections between the encoder and the decoder of a VAE are used to reproduce details, additional training in image space will be required, limiting the application to tasks in latent space such as image stylization. In this work, we propose a pixel-aware stable diffusion (PASD) network to achieve robust Real-ISR and personalized image stylization. Specifically, a pixel-aware cross attention module is introduced to enable diffusion models perceiving image local structures in pixel-wise level, while a degradation removal module is used to extract degradation insensitive features to guide the diffusion process together with image high level information. An adjustable noise schedule is introduced to further improve the image restoration results. By simply replacing the base diffusion model with a stylized one, PASD can generate diverse stylized images without collecting pairwise training data, and by shifting the base model with an aesthetic one, PASD can bring old photos back to life. Extensive experiments in a variety of image enhancement and stylization tasks demonstrate the effectiveness of our proposed PASD approach. Our source codes are available at \url{https://github.com/yangxy/PASD/}.

Figures (13)

• Figure 1: From left to right: an input LQ image, the Real-ISR outputs by ControlNet zhang2023controlnet and our PASD method. One can see that the output by ControlNet has clear content inconsistency with the input, while PASD preserves the structure well in pixel level.
• Figure 2: Architecture of the proposed pixel-aware stable diffusion (PASD) network. PASD consists of several modules, including Degradation Removal, ControlNet, PACA, ANS, and High-level Nets. During training, the encoder maps the HQ image to a latent representation, which is then added by noise to yield a noisy latent. In testing, the LQ image is used to generate the noisy input, while ANS is employed for flexible perception-fidelity balance. The noisy latent is fed to the UNet along with timestep, high-level information, and the output of ControlNet after Degradation Removal conditioned on the LQ image. In particular, the output of ControlNet is added to the UNet via PACA in latent space. PASD can be readily used for personalized stylization by simply switching the base diffusion model to a personalized one.
• Figure 3: The curves of PSNR/QAlign versus $\bar{\alpha}_a$.
• Figure 4: Visual results with different values of $\bar{\alpha}_a$.
• Figure 5: Realistic image super-resolution results by different methods.