Scale Where It Matters: Training-Free Localized Scaling for Diffusion Models

TL;DR

This paper presents LoTTS, a training-free localized test-time scaling framework for diffusion-based text-to-image generation that concentrates compute on defect-prone regions identified via quality-aware cross-attention prompts. It couples defect localization with consistency-maintenance through localized noise injection and denoising, ensuring refined regions harmonize with the global content. Empirically, LoTTS delivers state-of-the-art improvements in local quality and global fidelity across SD2.1, SDXL, and FLUX while achieving substantial GPU-cost reductions (roughly two to fourfold) compared with Best-of-N global sampling. The work establishes localized TTS as a practical, architecture-agnostic strategy for inference-time scaling and provides theoretical and empirical insights into when region-focused refinements outperform global approaches.

Abstract

Diffusion models have become the dominant paradigm in text-to-image generation, and test-time scaling (TTS) further improves quality by allocating more computation during inference. However, existing TTS methods operate at the full-image level, overlooking the fact that image quality is often spatially heterogeneous. This leads to unnecessary computation on already satisfactory regions and insufficient correction of localized defects. In this paper, we explore a new direction - Localized TTS - that adaptively resamples defective regions while preserving high-quality regions, thereby substantially reducing the search space. This paradigm poses two central challenges: accurately localizing defects and maintaining global consistency. We propose LoTTS, the first fully training-free framework for localized TTS. For defect localization, LoTTS contrasts cross- and self-attention signals under quality-aware prompts (e.g., high-quality vs. low-quality) to identify defective regions, and then refines them into coherent masks. For consistency, LoTTS perturbs only defective regions and denoises them locally, ensuring that corrections remain confined while the rest of the image remains undisturbed. Extensive experiments on SD2.1, SDXL, and FLUX demonstrate that LoTTS achieves state-of-the-art performance: it consistently improves both local quality and global fidelity, while reducing GPU cost by 2-4x compared to Best-of-N sampling. These findings establish localized TTS as a promising new direction for scaling diffusion models at inference time.

Figures (16)

• Figure 1: Global vs. Local Test-Time Scaling. Conventional TTS methods perform global search, sampling or perturbing the entire image, which ignores the inherent spatial heterogeneity of image quality and wastes computation on regions that are already good. LoTTS instead performs localized refinement: it identifies defective regions using quality-aware masks and selectively resamples only where needed, improving low-quality areas while preserving high-quality content.
• Figure 1: Additional examples of mask generation in SD2.1. LoTTS produces reliable quality-aware masks across diverse prompts, effectively localizing low-quality regions for refinement.
• Figure 2: Overview of LoTTS. LoTTS begins by generating multiple candidate images from different noise seeds, identifies defective regions via high-/low-quality prompt contrast, constructs a quality-aware defect mask, injects noise selectively within the masked areas, performs localized denoising with spatial and temporal consistency, and finally selects the best refined sample with a verifier. This pipeline selectively improves low-quality regions while preserving global fidelity.
• Figure 2: Additional examples of mask generation in FLUX. LoTTS produces finer and more detailed masks in FLUX, consistently localizing local degradations under varied prompts and enabling more precise localized resampling.
• Figure 3: LoTTS Framework for Defect Localization. The pipeline consists of four stages: prompt-driven discrimination, context-aware propagation, semantic-guided reweighting, and quality-aware mask generation.

Theorems & Definitions (12)

• Lemma 1: Expected TP/FP Statistics
• proof
• Theorem 1: Expected Quality Gains
• Corollary 1: General Case
• Corollary 2: Dominance Under Cost Asymmetry
• proof
• Corollary 3: Dominance Under Equal-Cost Sparse Regime
• proof
• Corollary 4: Saturation of Best-of-$N$ Global Resampling
• proof