CritiFusion: Semantic Critique and Spectral Alignment for Faithful Text-to-Image Generation

TL;DR

CritiFusion introduces a training-free, inference-time refinement for text-to-image diffusion that combines CritiCore’s multimodal semantic critique with SpecFusion’s spectral-domain consistency. CritiCore leverages a Vision-Language Model and a Multi-LLM committee to produce a refined conditioning $\tilde{c}$, while SpecFusion merges low-frequency structure from the base pass with high-frequency details from the refined pass, preserving layout and illumination. The approach is backbone-agnostic and demonstrates significant gains in semantic alignment and perceptual quality across SD v1.5 and SDXL backbones, without any backpropagation or retraining. With qualitative and quantitative improvements on standard prompts and metrics, CritiFusion offers a scalable plug-in refinement that complements existing diffusion models and other inference-time methods, potentially extending to video and 3D generation.

Abstract

Recent text-to-image diffusion models have achieved remarkable visual fidelity but often struggle with semantic alignment to complex prompts. We introduce CritiFusion, a novel inference-time framework that integrates a multimodal semantic critique mechanism with frequency-domain refinement to improve text-to-image consistency and detail. The proposed CritiCore module leverages a vision-language model and multiple large language models to enrich the prompt context and produce high-level semantic feedback, guiding the diffusion process to better align generated content with the prompt's intent. Additionally, SpecFusion merges intermediate generation states in the spectral domain, injecting coarse structural information while preserving high-frequency details. No additional model training is required. CritiFusion serves as a plug-in refinement stage compatible with existing diffusion backbones. Experiments on standard benchmarks show that our method notably improves human-aligned metrics of text-to-image correspondence and visual quality. CritiFusion consistently boosts performance on human preference scores and aesthetic evaluations, achieving results on par with state-of-the-art reward optimization approaches. Qualitative results further demonstrate superior detail, realism, and prompt fidelity, indicating the effectiveness of our semantic critique and spectral alignment strategy.

Figures (13)

• Figure 1: Diverse high-fidelity generations by our CritiFusion model. Our method synthesizes diverse artistic and photorealistic scenes with improved prompt alignment and high visual fidelity.
• Figure 2: Overview of our framework. Our pipeline has two modules. (1) CritiCore fuses VLM captions with Multi-LLM feedback to produce a refined prompt embedding $\tilde{c}$. A diffusion backbone samples a base latent $\mathbf{z}^{\text{base}}$ (decoded as $\mathbf{x}^{\text{base}}$) and, guided by $\tilde{c}$, partially re-denoises it to $\mathbf{z}^{\text{ref}}$. (2) SpecFusion performs frequency gating: the high-frequency spectrum of $\mathbf{z}^{\text{ref}}$ is combined with the low-frequency spectrum of $\mathbf{z}^{\text{base}}$ to yield $\tilde{\mathbf{z}}$, which is decoded to the final image $\hat{\mathbf{x}}$. This two-stage refinement corrects semantic misalignment, enriches detail, and preserves global coherence.
• Figure 3: Qualitative comparison on various prompts. We compare images generated by SDXL, FLUX, Diffusion-DPO, SPO, DyMO, and our CritiFusion method across various subjects such as architecture, animals, natural scenes, and characters. Each group shares the same prompt for fairness. Our results (far right in each group) show superior realism and semantic alignment: natural color tones, spatial coherence, and faithful prompt adherence. Competing methods often suffer from artifacts or miss subtle details, while CritiFusion maintains balanced composition and high-fidelity rendering. Zoom-in reveals texture and object accuracy, illustrating the benefits of semantic critique and spectral fusion.
• Figure 4: Plug-in enhancement on SOTA models. Each pair compares the original outputs (left) and the corresponding results after applying CritiFusion (right). CritiFusion consistently improves semantic alignment and visual fidelity while preserving the original style.
• Figure 5: Qualitative ablations. Each row shows the outputs from the baseline SDXL, +Multi-LLM, +VLM, and the full CritiFusion model. Adding each component progressively enhances semantic correctness and visual fidelity.