Guiding Token-Sparse Diffusion Models

TL;DR

Sparse Guidance introduces a finetune-free, test-time token sparsity signal for sparsely trained diffusion models, addressing the inability of CFG to effectively guide such models during inference. By using two sparsity levels to create a capacity gap, SG steers conditional predictions toward higher-quality, more diverse outputs with lower compute. Empirical results on ImageNet-256 achieve $\text{FID}=1.58$ with $25\%$ fewer FLOPs, and a 2.5B text-to-image model shows improved human preference and higher throughput. The approach scales to large T2I models, improves sample diversity, and reduces inference cost, offering a practical path to deploy token-sparse diffusion without dense finetuning.

Abstract

Diffusion models deliver high quality in image synthesis but remain expensive during training and inference. Recent works have leveraged the inherent redundancy in visual content to make training more affordable by training only on a subset of visual information. While these methods were successful in providing cheaper and more effective training, sparsely trained diffusion models struggle in inference. This is due to their lacking response to Classifier-free Guidance (CFG) leading to underwhelming performance during inference. To overcome this, we propose Sparse Guidance (SG). Instead of using conditional dropout as a signal to guide diffusion models, SG uses token-level sparsity. As a result, SG preserves the high-variance of the conditional prediction better, achieving good quality and high variance outputs. Leveraging token-level sparsity at inference, SG improves fidelity at lower compute, achieving 1.58 FID on the commonly used ImageNet-256 benchmark with 25% fewer FLOPs, and yields up to 58% FLOP savings at matched baseline quality. To demonstrate the effectiveness of Sparse Guidance, we train a 2.5B text-to-image diffusion model using training time sparsity and leverage SG during inference. SG achieves improvements in composition and human preference score while increasing throughput at the same time.

Figures (18)

• Figure 1: Sparse Guidance provides effective, efficient, structure-preserving guidance for sparsely trained diffusion models. (Left) Unlike Classifier-free Guidance, SG stays closer to the conditional prediction, yielding higher-variance, non-collapsed samples. (Right, top) On ImageNet-256, SG (Quality) attains an FID of 1.58 without any previously required dense finetuning while also increasing throughput, and SG (Speed) matches the baseline quality at substantially lower inference cost. (Right, bottom) Applied to our 2.5B text-to-image model, Sparse Guidance raises its HPSv3 ma2025hpsv3 performance enough to surpass a range of larger models, which it could not achieve without SG.
• Figure 2: Classifier-free Guidance (CFG) provides limited benefits for token-sparse diffusion models. While token-sparse training produces stronger conditional diffusion models than standard dense training, their practical impact has been constrained by poor compatibility with CFG, which limits inference quality and slows adoption in practice. Sparse Guidance (SG) overcomes this limitation, restoring strong guidance gains for token-sparse models and enabling them to match or surpass the image quality of their dense baselines.
• Figure 3: Masking and Routing as two types of token-level sparsity. Masking replaces tokens with learnable mask token zheng2023fast_maskdit while routing preserves information by reintroducing tokens krause2025tread.
• Figure 4: Without Sparse Guidance, image quality and composition worsens consistently with increased token-sparsity ratios.
• Figure 5: Sparse Guidance improves both convergence and training-time sample quality for sparsely trained diffusion models.Left FID over training iterations comparing CFG, CFG with dense finetuning, and Sparse Guidance (SG). where SG achieves the lowest FID using the best CFG scale $\omega$ for each method. Right Training-time sample progress using SG, showing that sparsely trained models already produce high-fidelity samples without an additional dense finetuning stage, enabling direct visual evaluation during training.