Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FocusDiff: Advancing Fine-Grained Text-Image Alignment for Autoregressive Visual Generation through RL

Kaihang Pan, Wendong Bu, Yuruo Wu, Yang Wu, Kai Shen, Yunfei Li, Hang Zhao, Juncheng Li, Siliang Tang, Yueting Zhuang

TL;DR

The paper addresses the challenge of fine-grained text-image alignment in autoregressive text-to-image generation by introducing the PairComp benchmark, which tests models on paired prompts with subtle semantic differences. It proposes FocusDiff, comprising FocusDiff-Data and Pair-GRPO, to learn from differences between similar text-image pairs via a QA-based reward and an expanded group policy that balances exploration and exploitation. The approach yields state-of-the-art results on existing text-to-image benchmarks and substantial gains on PairComp, demonstrating improved control over visual tokens and robustness to counterfactual prompts. This work advances precise multimodal alignment in AR-based generation and offers a scalable framework for training data and RL-based refinement.

Abstract

Recent studies extend the autoregression paradigm to text-to-image generation, achieving performance comparable to diffusion models. However, our new PairComp benchmark -- featuring test cases of paired prompts with similar syntax but different fine-grained semantics -- reveals that existing models struggle with fine-grained text-image alignment thus failing to realize precise control over visual tokens. To address this, we propose FocusDiff, which enhances fine-grained text-image semantic alignment by focusing on subtle differences between similar text-image pairs. We construct a new dataset of paired texts and images with similar overall expressions but distinct local semantics, further introducing a novel reinforcement learning algorithm to emphasize such fine-grained semantic differences for desired image generation. Our approach achieves state-of-the-art performance on existing text-to-image benchmarks and significantly outperforms prior methods on PairComp.

FocusDiff: Advancing Fine-Grained Text-Image Alignment for Autoregressive Visual Generation through RL

TL;DR

The paper addresses the challenge of fine-grained text-image alignment in autoregressive text-to-image generation by introducing the PairComp benchmark, which tests models on paired prompts with subtle semantic differences. It proposes FocusDiff, comprising FocusDiff-Data and Pair-GRPO, to learn from differences between similar text-image pairs via a QA-based reward and an expanded group policy that balances exploration and exploitation. The approach yields state-of-the-art results on existing text-to-image benchmarks and substantial gains on PairComp, demonstrating improved control over visual tokens and robustness to counterfactual prompts. This work advances precise multimodal alignment in AR-based generation and offers a scalable framework for training data and RL-based refinement.

Abstract

Recent studies extend the autoregression paradigm to text-to-image generation, achieving performance comparable to diffusion models. However, our new PairComp benchmark -- featuring test cases of paired prompts with similar syntax but different fine-grained semantics -- reveals that existing models struggle with fine-grained text-image alignment thus failing to realize precise control over visual tokens. To address this, we propose FocusDiff, which enhances fine-grained text-image semantic alignment by focusing on subtle differences between similar text-image pairs. We construct a new dataset of paired texts and images with similar overall expressions but distinct local semantics, further introducing a novel reinforcement learning algorithm to emphasize such fine-grained semantic differences for desired image generation. Our approach achieves state-of-the-art performance on existing text-to-image benchmarks and significantly outperforms prior methods on PairComp.

Paper Structure

This paper contains 31 sections, 2 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Benchmark: PairComp
  3. Data format and Task Categorization.
  4. Evaluation Protocols.
  5. Method: FocusDiff
  6. Data Perspective: FocusDiff-Data
  7. Training Perspective: Pair-GRPO
  8. QA-based Reward.
  9. Vanilla GRPO for Autoregressive Image Generation.
  10. Pair-GRPO for Fine-Grained Semantic Focusing.
  11. Experiments
  12. Main Results on PairComp
  13. Main Results on Existing Benchmarks
  14. Qualitative Comparisons
  15. Effect of Pair-GRPO.
  16. ...and 16 more sections

Figures (8)

  • Figure 1: (a) For Janus-Pro-7B, the geometric mean score in PairComp is significantly lower than the arithmetic mean score (b) Examples of Janus-Pro-7B failing to generate images precisely according to the prompt. (c) The subtle sensory differences between images or between texts result in only minor alterations to specific tokens.
  • Figure 2: Statistical information of PairComp and test case examples for each subtask.
  • Figure 3: The framework of our Pair-GRPO.
  • Figure 4: Qualitative Comparisons between Janus-Pro-7B and our Janus-FocusDiff on pairs of similar prompts. For each prompt, we ask each model to generate two images.
  • Figure 5: More qualitative Comparisons between Janus-Pro-7B and Janus-FocusDiff on pairs of similar prompts.
  • ...and 3 more figures