SIDiffAgent: Self-Improving Diffusion Agent

TL;DR

SIDiffAgent tackles prompt sensitivity and artifact generation in diffusion-based image synthesis by introducing a training-free, multi-agent framework that leverages memory and Theory-of-Mind–inspired coordination. It comprises a Generation Orchestrator, Evaluation, and Guidance module, with a persistent experience memory that retrieves past trajectories to inject prompt-based cues at run time, guided by threshold-based quality checks $\tau$. At test time, the Evaluation Agent computes aesthetics and text–image alignment scores and triggers iterative edits via Qwen-Image-Edit to refine outputs. Empirically, SIDiffAgent achieves state-of-the-art VQA scores on GenAIBench and DrawBench and outperforms both proprietary and open-source baselines, demonstrating a practical, training-free path to reliable and controllable diffusion outputs for real-world tasks. These results highlight the potential of test-time agentic coordination and memory-based guidance for scalable diffusion-based generation without retraining.

Abstract

Text-to-image diffusion models have revolutionized generative AI, enabling high-quality and photorealistic image synthesis. However, their practical deployment remains hindered by several limitations: sensitivity to prompt phrasing, ambiguity in semantic interpretation (e.g., ``mouse" as animal vs. a computer peripheral), artifacts such as distorted anatomy, and the need for carefully engineered input prompts. Existing methods often require additional training and offer limited controllability, restricting their adaptability in real-world applications. We introduce Self-Improving Diffusion Agent (SIDiffAgent), a training-free agentic framework that leverages the Qwen family of models (Qwen-VL, Qwen-Image, Qwen-Edit, Qwen-Embedding) to address these challenges. SIDiffAgent autonomously manages prompt engineering, detects and corrects poor generations, and performs fine-grained artifact removal, yielding more reliable and consistent outputs. It further incorporates iterative self-improvement by storing a memory of previous experiences in a database. This database of past experiences is then used to inject prompt-based guidance at each stage of the agentic pipeline. \modelour achieved an average VQA score of 0.884 on GenAIBench, significantly outperforming open-source, proprietary models and agentic methods. We will publicly release our code upon acceptance.

Figures (9)

• Figure 1: Qualitative comparison between our model, SIDiffAgent, and the state-of-the-art, Qwen-Image model. SIDiffAgent demonstrates superior text-to-image alignment and realism, accurately rendering prompts that require nuanced understanding. Notably, our model successfully handles: negative constraints ('no umbrellas'), complex scene composition, anthropomorphic roles ('athlete cat'), and precise object counting.
• Figure 2: Workflow of the Self-Improving Diffusion Agent (SIDiffAgent). An input prompt is processed by the Generation Orchestrator Agent ($A_{\text{ORC}}$), which employs sub-agents ($S_{\text{CRE}}$, $S_{\text{INT}}$, $S_{\text{REF}}$, $S_{\text{NEG}}$) to assess creativity, clarify intent, refine the prompt, and add adaptive negative constraints before generation ($S_{\text{GEN}}$). The Evaluation Agent ($A_{\text{EVAL}}$) scores the generated image on aesthetic quality and text–image alignment, triggering targeted refinements if the evaluation score is less than the pre-defined threshold $\tau$ . Each trajectory is stored in the knowledge base, where the Guidance Agent ($A_{\text{GUID}}$) stores pitfalls and successes into corrective and workflow guidance, which are injected back into decision nodes to improve future generations.
• Figure 3: Qualitative comparison of SIDiffAgent with 10 models on prompts from DrawBench (Top) and GenAI-Bench (Bottom).SIDiffAgent demonstrates superior handling of spatial relationships (correctly placing the sheep 'to the right of' the glass) and compositional complexity (generating all elements of the cat, person, and fireplace scene) compared to other models.
• Figure 4: The first row shows images generated by T2I-Copilot without memory, while the second row shows outputs from our approach. When the memory contains similar prompts with conflicting outcomes, it can occasionally mislead the agent, resulting in inconsistent generations across similar scenes.
• Figure 5: Comparison between initially generated and regenerated images. Left: In complex scenes with intricate textures and motion-like artifacts, iterative regeneration amplifies some visual details while degrading others, leading to unstable convergence across iterations. Right: When the creativity agent suggests uncommon or compositionally difficult attributes, the generator repeatedly fails to realize them, triggering unnecessary correction loops and producing minimal meaningful improvement over successive regenerations.