SRSR: Enhancing Semantic Accuracy in Real-World Image Super-Resolution with Spatially Re-Focused Text-Conditioning

TL;DR

This work tackles semantic misalignment in diffusion-based real-world image super-resolution by introducing SRSR, a plug-and-play, inference-time framework with two key components. Spatially Re-focused Cross-Attention (SRCA) grounds each text token to a corresponding image region, masking out irrelevant regions to reduce semantic drift, while Spatially Targeted Classifier-Free Guidance (STCFG) applies unconditional guidance to ungrounded areas to prevent hallucinations. The method leverages DAPE for degradation-aware prompt extraction and Grounded SAM for precise grounding, enabling strong improvements in fidelity (PSNR/SSIM) and perceptual quality (LPIPS/DISTS) on both synthetic and real-world datasets, with state-of-the-art performance on several benchmarks. Importantly, SRSR is inference-only and model-agnostic, preserving parameter counts and allowing seamless integration with existing cross-attention-based SR pipelines. These advances have practical impact for Real-ISR tasks requiring semantically faithful restorations and open-vocabulary grounding, while highlighting the need for robust grounding and evaluation metrics for no-reference scenarios.

Abstract

Existing diffusion-based super-resolution approaches often exhibit semantic ambiguities due to inaccuracies and incompleteness in their text conditioning, coupled with the inherent tendency for cross-attention to divert towards irrelevant pixels. These limitations can lead to semantic misalignment and hallucinated details in the generated high-resolution outputs. To address these, we propose a novel, plug-and-play spatially re-focused super-resolution (SRSR) framework that consists of two core components: first, we introduce Spatially Re-focused Cross-Attention (SRCA), which refines text conditioning at inference time by applying visually-grounded segmentation masks to guide cross-attention. Second, we introduce a Spatially Targeted Classifier-Free Guidance (STCFG) mechanism that selectively bypasses text influences on ungrounded pixels to prevent hallucinations. Extensive experiments on both synthetic and real-world datasets demonstrate that SRSR consistently outperforms seven state-of-the-art baselines in standard fidelity metrics (PSNR and SSIM) across all datasets, and in perceptual quality measures (LPIPS and DISTS) on two real-world benchmarks, underscoring its effectiveness in achieving both high semantic fidelity and perceptual quality in super-resolution.

Figures (22)

• Figure 1: Illustrations of how inherent cross-attention can be misled by irrelevant tokens, resulting in semantically incorrect restorations (top). Left: the baseline mistakenly associates the stone region with the token 'bird', and the animal’s neck and beak with 'stone', producing wing-like artifacts on the stone and unnatural textures on the animal. Right: attention for 'stare' is scattered across irrelevant patches, and both the eye and lion face incorrectly respond to 'grass', introducing hallucinated textures. We propose re-focusing cross-attention by constraining the influence of each text token to its grounded region, yielding sharper and semantically aligned reconstructions (bottom).
• Figure 2: Analysis of how our proposed Spatially Re-focused Cross-Attention (SRCA) and Spatially Targeted Classifier-Free Guidance (STCFG) improve semantic fidelity in text-conditioned super-resolution. The Ungrounded mask highlights regions where no textual tag can be confidently grounded. Existing methods rely solely on inherent cross-attention, allowing global or irrelevant tokens to influence all regions. SRCA addresses this by limiting each token’s influence to its corresponding grounded region only, reducing semantic confusion. However, this leaves ungrounded regions only associated with global tokens (e.g., EOS, padding, punctuation), where summary tokens like EOS can still carry semantics of the entire prompt and influence its restoration. To resolve this, STCFG disables text conditioning entirely in ungrounded areas by using unconditional noise prediction in the reverse diffusion process, further enhancing the ungrounded region's restoration.
• Figure 3: Overview of our proposed pipeline. First, the LR image is processed by a Degradation-Aware Prompt Extractor (DAPE) seesr to obtain text tags. Both the LR image and the extracted tags are then passed to Grounded SAM, which produces visually grounded tag--mask pairs. We also define an ungrounded mask as the complement of the union of all grounded masks. Next, each tag--mask pair is integrated into all 16 U-Net cross-attention layers, using the masks to constrain text conditioning precisely to relevant regions. Once noise prediction is complete, we selectively apply Classifier-Free Guidance (CFG) to grounded pixels while leaving ungrounded pixels under unconditional guidance, ensuring they remain unaffected by the text prompts. After $T$ denoising steps, a final decoder maps the latent representation back to pixel space, yielding the super-resolved (SR) image.
• Figure 4: Qualitative results show that the baseline method exhibits clear semantic hallucinations. In contrast, plugging SRSR into the baseline leads to semantically faithful restorations. All full-reference metrics, including both fidelity (PSNR and SSIM) and perceptual quality (LPIPS and DISTS) measures, consistently validate the improvements brought by our method. However, it is worth noting that the no-reference metrics tend to misjudge and heavily reward hallucinated results due to their design, as indicated by the significant performance gap observed despite degraded semantic realism. This exposes their limitations in assessing semantic fidelity for super-resolution.
• Figure S1: Column 1: Paired LR-HR images. Column 2: Qualitative comparisons of the baseline with and without our proposed SRSR. Column 3: Corresponding quantitative metrics. Columns 4–5: Attention visualizations for selected tokens to illustrate the effect of SRSR. In the region highlighted by the red bounding box, the object is actually an animal’s claw, but it is difficult to recognize in the degraded LR image. Consequently, the prompt extractor (DAPE) fails to extract relevant tags such as 'claw’. As a result, the baseline model attributes this region to the token 'animal’ according to the cross-attention map, and hallucinates it as a vivid fish. Similarly, in the yellow bounding box, the region corresponds to the animal’s fur, but is misinterpreted due to degradation. The baseline instead applies influence from the unrelated tag 'stone’, resulting in a texture resembling small pebbles. Beyond these highlighted objects, the baseline also introduces other over-synthesized textures in the 'animal’ and 'stone’ regions that deviate from the ground-truth HR image, despite being visually plausible. In contrast, our SRSR framework assigns tags only to regions where grounding confidence is high (e.g., 'animal’, 'stone’), leaving uncertain regions, such as the red region, ungrounded. Within the SRSR framework, SRCA ensures that grounded regions are not influenced by irrelevant tokens, thereby removing hallucinations from the 'animal’ and 'stone’ areas. Additionally, STCFG applies unconditional predictions to ungrounded regions like the animal’s claw, suppressing inappropriate text influence while preserving perceptual quality.