Trust but Verify: Adaptive Conditioning for Reference-Based Diffusion Super-Resolution via Implicit Reference Correlation Modeling

TL;DR

Ada-RefSR, a single-step diffusion framework guided by a "Trust but Verify"principle, achieves a strong balance of fidelity, naturalness, and efficiency, while remaining robust under varying reference alignment.

Abstract

Recent works have explored reference-based super-resolution (RefSR) to mitigate hallucinations in diffusion-based image restoration. A key challenge is that real-world degradations make correspondences between low-quality (LQ) inputs and reference (Ref) images unreliable, requiring adaptive control of reference usage. Existing methods either ignore LQ-Ref correlations or rely on brittle explicit matching, leading to over-reliance on misleading references or under-utilization of valuable cues. To address this, we propose Ada-RefSR, a single-step diffusion framework guided by a "Trust but Verify" principle: reference information is leveraged when reliable and suppressed otherwise. Its core component, Adaptive Implicit Correlation Gating (AICG), employs learnable summary tokens to distill dominant reference patterns and capture implicit correlations with LQ features. Integrated into the attention backbone, AICG provides lightweight, adaptive regulation of reference guidance, serving as a built-in safeguard against erroneous fusion. Experiments on multiple datasets demonstrate that Ada-RefSR achieves a strong balance of fidelity, naturalness, and efficiency, while remaining robust under varying reference alignment.

Figures (20)

• Figure 1: Visual comparisons: S3Diff is a single-image generation network, and ReFIR is the current SOTA for reference based restoration. Our method not only outperforms ReFIR in leveraging reference details but also shows stronger robustness against degradations than S3Diff.
• Figure 2: Comparison of gating strategies. Prior methods either use non-interactive global gating (PFStorer) or depend on explicit token correlations that are easily disturbed by noise (ReFIR). In contrast, we introduce implicit, correlation-driven gating that selectively activates useful reference cues. "src tokens" denote main-branch features, "ref tokens" are raw reference features, and "ref-aligned fusion tokens" are reference features aligned to the source space through feature fusion. Here, $\mathbf{G}$ denotes the per-token gating values generated by our AICG, while $\mathbf{S}$ represents the full token-to-token similarity map used in explicit correlation-based gating methods like ReFIR.
• Figure 3: Overview of our framework. It comprises two components: (a) a reference-based restoration backbone, and (b) a correlation-aware adaptive gating mechanism.
• Figure 3: GPU Memory and Inference Time at Different Resolutions.
• Figure 4: Vanilla RA and ReFIR both introduce artifacts by overusing irrelevant reference regions (left: duplicate eye; right: cluttered background). Our gating mechanism suppresses these activations, leading to more natural results. Zoom in for better visualization.