F2IDiff: Real-world Image Super-resolution using Feature to Image Diffusion Foundation Model

TL;DR

This work introduces F2IDiff-SR, a real-world image super-resolution method that replaces text-based diffusion conditioning with low-level DINOv2 image features, enabling faithful reconstruction with minimal hallucinations for smartphone imagery. Trained on a compact dataset of roughly $3.8\times 10^4$ HR images, the Feature-to-Image Diffusion (F2IDiff) FM enables a lighter U-Net (Eff-F2IDiff) and, when combined with LoRA-based SR networks, delivers superior PSNR, SSIM, and FID on RealSR and DRealSR benchmarks compared with public T2IDiff-based SR methods. The approach also demonstrates strong performance on synthetic DIV2K-like data, while avoiding the heavy computational burden of billion-image pretraining. Overall, F2IDiff-SR achieves higher fidelity and fewer hallucinations in real-world smartphone SR scenarios and reduces model complexity, offering a practical path for deployable real-world SISR. Future work proposes expanding the HR training set toward 100k images and refining the FM’s VAE to further enhance controllable, hallucination-free generation.

Abstract

With the advent of Generative AI, Single Image Super-Resolution (SISR) quality has seen substantial improvement, as the strong priors learned by Text-2-Image Diffusion (T2IDiff) Foundation Models (FM) can bridge the gap between High-Resolution (HR) and Low-Resolution (LR) images. However, flagship smartphone cameras have been slow to adopt generative models because strong generation can lead to undesirable hallucinations. For substantially degraded LR images, as seen in academia, strong generation is required and hallucinations are more tolerable because of the wide gap between LR and HR images. In contrast, in consumer photography, the LR image has substantially higher fidelity, requiring only minimal hallucination-free generation. We hypothesize that generation in SISR is controlled by the stringency and richness of the FM's conditioning feature. First, text features are high level features, which often cannot describe subtle textures in an image. Additionally, Smartphone LR images are at least $12MP$, whereas SISR networks built on T2IDiff FM are designed to perform inference on much smaller images ($<1MP$). As a result, SISR inference has to be performed on small patches, which often cannot be accurately described by text feature. To address these shortcomings, we introduce an SISR network built on a FM with lower-level feature conditioning, specifically DINOv2 features, which we call a Feature-to-Image Diffusion (F2IDiff) Foundation Model (FM). Lower level features provide stricter conditioning while being rich descriptors of even small patches.

Figures (5)

• Figure 1: In top figure, F2IDiff-SR shows exceptional performance on the Real-SISR ($4\times$) task, achieving better metrics:- PSNR, SSIM, and FID. It outperforms SOTA methods by a substantial margin, highlighting its effectiveness. In the bottom figure, F2IDiff-SR gives better results compared to other methods on real-world images captured from an S25 Ultra smartphone. The other methods generate inconsistent texture and hallucinations.
• Figure 2: Zoom-in for details: SOTA single-step diffusion SR methods, such as OSEDiff OSEDiff and PiSA-SR PiSASR, produce excessive hallucination on real-world test datasets. For instance, OSEDiff generates a bird's tail in the output image even when no bird is present in the input. Similarly, PiSA-SR produces fur instead of facial features, leading to inaccurate and unrealistic results.
• Figure 3: Our Methods:(a) Training pipeline of T2IDiff FM: A diffusion U-Net with text conditioning is trained from scratch on internal 38K HR images using a pre-trained Encoder-Decoder, Florence caption generation, and text-encoder. (b) Training pipeline of F2IDiff FM: A diffusion U-Net using DINOv2 features as conditioning is trained from scratch on internal 38K HR images using a pre-trained Encoder-Decoder, DINOv2 feature extractor. (c) SISR network based on T2IDiff FM: A single-step diffusion SR model is built on T2IDiff FM using LoRA. (d) SISR network based on F2IDiff FM: A single-step diffusion SR model is built on F2IDiff FM using LoRA.
• Figure 4: Zoom-in for best visuals: Qualitative comparison between our methods on public datasets and smartphone captured images. (a) Our method on public datasets show the highest fidelity with HR, while OSEDiff OSEDiff, which uses public SDV2.1 Rombach_2022_CVPR_SD2.1, generates artificial textures. (b) Our method generates uniform textures and natural textures compared to other methods which use public SDV2.1 Rombach_2022_CVPR_SD2.1.
• Figure 5: Zoom-in for best visuals: Qualitative comparisons of our methods (F2IDiff-SR, Eff-F2IDiff-SR) with SOTA GAN-based methods, multi-step diffusion methods, and single-step diffusion methods. The SOTA diffusion methods show significant hallucinations and unrealistic texture. Our methods generate details while preserving best fidelity.