Iterative Inference-time Scaling with Adaptive Frequency Steering for Image Super-Resolution

TL;DR

Diffusion models offer strong SR capabilities but struggle to simultaneously maximize perceptual detail and structural fidelity. The authors introduce IAFS, a training-free framework that iteratively refines samples and adaptively steers frequency content by fusing high-frequency perceptual cues with low-frequency structural information. By employing a segmented, iteration-aware reward schedule and Adaptive Frequency Steering (AFS) via AdaIN-aligned low-frequency references, IAFS delivers superior perceptual quality and fidelity across multiple diffusion SR backbones with modest overhead. Extensive experiments on ImageNet, DIV2K, RealSR, and DRealSR demonstrate consistent gains over existing inference-time scaling methods, highlighting the method’s practical impact and potential applicability to broader image-generation tasks.

Abstract

Diffusion models have become a leading paradigm for image super-resolution (SR), but existing methods struggle to guarantee both the high-frequency perceptual quality and the low-frequency structural fidelity of generated images. Although inference-time scaling can theoretically improve this trade-off by allocating more computation, existing strategies remain suboptimal: reward-driven particle optimization often causes perceptual over-smoothing, while optimal-path search tends to lose structural consistency. To overcome these difficulties, we propose Iterative Diffusion Inference-Time Scaling with Adaptive Frequency Steering (IAFS), a training-free framework that jointly leverages iterative refinement and frequency-aware particle fusion. IAFS addresses the challenge of balancing perceptual quality and structural fidelity by progressively refining the generated image through iterative correction of structural deviations. Simultaneously, it ensures effective frequency fusion by adaptively integrating high-frequency perceptual cues with low-frequency structural information, allowing for a more accurate and balanced reconstruction across different image details. Extensive experiments across multiple diffusion-based SR models show that IAFS effectively resolves the perception-fidelity conflict, yielding consistently improved perceptual detail and structural accuracy, and outperforming existing inference-time scaling methods.

Figures (7)

• Figure 1: Comparison of different inference-time scaling strategies. (a) Diffusion inference-time scaling based on Particle Optimization. (b) Diffusion inference-time scaling based on Optimal Path Search. (c) Iterative Diffusion inference-time scaling based on Adaptive Frequency Steering (IAFS).
• Figure 2: The overall architecture of Iterative Inference-Time Scaling with Adaptive Frequency Steering. At each iteration, the diffusion output serves as pseudo-GT for the next one. For each timestep, $N$ particles are sampled and the chosen particle is selected via hybrid reward $R$. The top-$K$ most similar particles are averaged as the reference. The chosen particle is then decomposed into high/low-frequency components, aligned via Adaptive Instance Normalization (AdaIN), and fused to produce the optimal particle for the next timestep.
• Figure 3: Qualitative comparison of 4$\times$ Super-Resolution with different inference-time scaling methods on baselines.
• Figure 4: Influence of the number of sampled particles ($N$) and the number of reference particles ($K$) on super-resolution performance. The perceptual metrics (CLIPIQA and LPIPS) and structural metrics (PSNR and SSIM) are plotted against different values of $N$ under three reference-particle settings ($K=2, 3, 4$).
• Figure 5: Evolution of the 1D Power Spectral Density (PSD) during the reverse diffusion process.