CLIP-SR: Collaborative Linguistic and Image Processing for Super-Resolution

TL;DR

The paper tackles semantic misalignment and detail loss in extreme downsampling by fusing textual semantics with visual cues for large-factor SR up to 16$ imes$. It introduces a multi-modal pipeline featuring a Text-Image Fusion Block (TIFBlock), a Prompt Predictor, and an Iterative Refinement Module, all guided by CLIP features and enforced by a CLIP-based discriminator together with reconstruction, perceptual, and text-constrained adversarial losses. Key contributions include the TIFBlock design, iterative cross-modal fusion, and prompt-driven text guidance, validated on COCO, CUB, and CelebA with qualitative and quantitative gains and demonstrated editability via prompts. The approach offers a scalable, controllable path to semantically coherent SR, enabling realistic texture recovery and semantic consistency in challenging downsampling scenarios, while highlighting the need to handle ambiguous natural language prompts more robustly in future work.

Abstract

Convolutional Neural Networks (CNNs) have significantly advanced Image Super-Resolution (SR), yet most CNN-based methods rely solely on pixel-based transformations, often leading to artifacts and blurring, particularly under severe downsampling rates (\eg, 8$\times$ or 16$\times$). The recently developed text-guided SR approaches leverage textual descriptions to enhance their detail restoration capabilities but frequently struggle with effectively performing alignment, resulting in semantic inconsistencies. To address these challenges, we propose a multi-modal semantic enhancement framework that integrates textual semantics with visual features, effectively mitigating semantic mismatches and detail losses in highly degraded low-resolution (LR) images. Our method enables realistic, high-quality SR to be performed at large upscaling factors, with a maximum scaling ratio of 16$\times$. The framework integrates both text and image inputs using the prompt predictor, the Text-Image Fusion Block (TIFBlock), and the Iterative Refinement Module, leveraging Contrastive Language-Image Pretraining (CLIP) features to guide a progressive enhancement process with fine-grained alignment. This synergy produces high-resolution outputs with sharp textures and strong semantic coherence, even at substantial scaling factors. Extensive comparative experiments and ablation studies validate the effectiveness of our approach. Furthermore, by leveraging textual semantics, our method offers a degree of super-resolution editability, allowing for controlled enhancements while preserving semantic consistency. The code is available at https://github.com/hengliusky/CLIP-SR.

Figures (10)

• Figure 1: Visualization of the results recovered by our method from low-resolution (LR) inputs. We highlight the semantic coherence part by aligning the textual guidance with the high-resolution (HR) ground truth.
• Figure 2: An example comparison between the $16\times$ image SR results of our method and two SOTA SR methods: DICGAN ma2020deep and CRAFT li2023feature. Here, DICGAN and CRAFT are retrained on the same dataset as that used by our approach; LR is the input low-resolution image, and GT is the high-resolution ground truth (enlarged appropriately for visualization purposes).
• Figure 3: The architecture of our proposed multi-modal cooperative semantic enhancement model for large-factor image super-resolution (see subfigure (a)). Given an input low-resolution image $I_{LR}$ and text guidance $T$, features $F_{I}$ and $F_{T}$ are first extracted using an image encoder $E_{I}$ and a text encoder $E_{T}$, respectively. The text feature vector $F_{T}$ is further refined through a prompt predictor module $P$ and then processed by the CLIP-ViT model to enhance textual guidance. The refined text and image features are subsequently integrated using a Text-Image Fusion Block (TIFBlock), which aligns and combines the two modalities (see subfigure (b)). Within the TIFBlock, an affine transformation is applied in its text fusion module. This transformation employs two consecutive MLPs that generate channel-wise scaling parameters ($\hat{\gamma^{n}} = MLP_1(Softmax(F_{T}))$) and shifting parameters ($\hat{\beta^{n}} = MLP_2(Softmax(F_{T}))$). These parameters adaptively modulate the channel-wise features of the visual representation $\hat{F}_{I}^{n}$. Finally, the fused multi-modal features undergo iterative refinement and semantic enhancement through a continuous Conv-TIFBlock structure, which is referred to as an iterative refinement module $R$. This iterative process ensures that progressively improved super-resolution outputs with enhanced details and semantic coherence are obtained.
• Figure 4: Visual comparison among the $4\times$ SR results obtained with three SOTA SR methods, i.e., TGSR ma2022rethinking, XPSR qu2024xpsr, and TGESR gandikota2024text, and our method on the CUB dataset. Notably, $\#$ indicates the results reported in the corresponding original paper.
• Figure 5: Visual comparison among the $4\times$ SR results obtained with four SOTA SR methods, i.e., SuperFAN bulat2018super, DICGAN ma2020deep, XPSR qu2024xpsr, TGESR gandikota2024text, and our method on the CelebA dataset. $*$ denotes that 4$\times$ SR is applied on the basis of the settings of XPSR, where an input image with a 128$\times$128 resolution is upscaled to 512$\times$512.