Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VisionDirector: Vision-Language Guided Closed-Loop Refinement for Generative Image Synthesis

Meng Chu, Senqiao Yang, Haoxuan Che, Suiyun Zhang, Xichen Zhang, Shaozuo Yu, Haokun Gui, Zhefan Rao, Dandan Tu, Rui Liu, Jiaya Jia

TL;DR

The paper tackles the difficulty of following long, multi-goal prompts in generative image synthesis by introducing LongGoalBench (LGBench) to benchmark real-world, multi-goal tasks and VisionDirector, a training-free, vision-language director that decomposes prompts, guides edits, and verifies outcomes. A GRPO-based post-training refines the planner to shorten edit trajectories and improve alignment, yielding state-of-the-art results on GenEval and ImgEdit. The approach reveals concrete bottlenecks in typography, logo fidelity, and lighting during multi-goal editing, and demonstrates that a modular director can significantly improve consistency without retraining diffusion backbones. The work suggests a practical path toward professional-grade, goal-aware creative tooling and provides benchmarks and infrastructure for future research in goal-conditioned generation and verification.

Abstract

Generative models can now produce photorealistic imagery, yet they still struggle with the long, multi-goal prompts that professional designers issue. To expose this gap and better evaluate models' performance in real-world settings, we introduce Long Goal Bench (LGBench), a 2,000-task suite (1,000 T2I and 1,000 I2I) whose average instruction contains 18 to 22 tightly coupled goals spanning global layout, local object placement, typography, and logo fidelity. We find that even state-of-the-art models satisfy fewer than 72 percent of the goals and routinely miss localized edits, confirming the brittleness of current pipelines. To address this, we present VisionDirector, a training-free vision-language supervisor that (i) extracts structured goals from long instructions, (ii) dynamically decides between one-shot generation and staged edits, (iii) runs micro-grid sampling with semantic verification and rollback after every edit, and (iv) logs goal-level rewards. We further fine-tune the planner with Group Relative Policy Optimization, yielding shorter edit trajectories (3.1 versus 4.2 steps) and stronger alignment. VisionDirector achieves new state of the art on GenEval (plus 7 percent overall) and ImgEdit (plus 0.07 absolute) while producing consistent qualitative improvements on typography, multi-object scenes, and pose editing.

VisionDirector: Vision-Language Guided Closed-Loop Refinement for Generative Image Synthesis

TL;DR

The paper tackles the difficulty of following long, multi-goal prompts in generative image synthesis by introducing LongGoalBench (LGBench) to benchmark real-world, multi-goal tasks and VisionDirector, a training-free, vision-language director that decomposes prompts, guides edits, and verifies outcomes. A GRPO-based post-training refines the planner to shorten edit trajectories and improve alignment, yielding state-of-the-art results on GenEval and ImgEdit. The approach reveals concrete bottlenecks in typography, logo fidelity, and lighting during multi-goal editing, and demonstrates that a modular director can significantly improve consistency without retraining diffusion backbones. The work suggests a practical path toward professional-grade, goal-aware creative tooling and provides benchmarks and infrastructure for future research in goal-conditioned generation and verification.

Abstract

Generative models can now produce photorealistic imagery, yet they still struggle with the long, multi-goal prompts that professional designers issue. To expose this gap and better evaluate models' performance in real-world settings, we introduce Long Goal Bench (LGBench), a 2,000-task suite (1,000 T2I and 1,000 I2I) whose average instruction contains 18 to 22 tightly coupled goals spanning global layout, local object placement, typography, and logo fidelity. We find that even state-of-the-art models satisfy fewer than 72 percent of the goals and routinely miss localized edits, confirming the brittleness of current pipelines. To address this, we present VisionDirector, a training-free vision-language supervisor that (i) extracts structured goals from long instructions, (ii) dynamically decides between one-shot generation and staged edits, (iii) runs micro-grid sampling with semantic verification and rollback after every edit, and (iv) logs goal-level rewards. We further fine-tune the planner with Group Relative Policy Optimization, yielding shorter edit trajectories (3.1 versus 4.2 steps) and stronger alignment. VisionDirector achieves new state of the art on GenEval (plus 7 percent overall) and ImgEdit (plus 0.07 absolute) while producing consistent qualitative improvements on typography, multi-object scenes, and pose editing.
Paper Structure (22 sections, 1 equation, 5 figures, 7 tables)

This paper contains 22 sections, 1 equation, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. LGBench: LongGoal Benchmark
  3. Benchmark Overview
  4. Scale and coverage.
  5. Benchmark Construction
  6. Evaluation Results
  7. Discussion and Motivation
  8. VisionDirector
  9. Overview
  10. Training-Free Closed-Loop Control
  11. GRPO for Efficient Generation
  12. Experiments
  13. Evaluation Setup
  14. Results on Text-to-Image
  15. Results on Image-to-Image
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Distribution of goal types in LGBench for both T2I and I2I subsets. The benchmark balances additive, stylistic, and semantic directives, reflecting the multi-constraint nature of real design tasks.
  • Figure 2: Construction pipelines for LongGoalBench.
  • Figure 3: Workflow of VisionDirector. The planner interprets long instructions, decides between one-shot or staged execution, performs micro-grid sampling, verifies progress, and rolls back if needed.
  • Figure 4: I2I result comparison with other models.
  • Figure 5: Adaptive decision-making in VisionDirector. The VLM exhibits distinct behavioral phases based on task complexity. (a) Iteration steps increase from 1-3 to 5-7 after exceeding 15 goals. (b) One-shot execution preference ($>$85%) transitions to staged execution ($<$10%) beyond the critical threshold, demonstrating the system's adaptive control strategy.