ReasonGen-R1: CoT for Autoregressive Image generation models through SFT and RL

TL;DR

ReasonGen-R1 introduces a two-stage framework that injects chain-of-thought reasoning into autoregressive image generation via supervised fine-tuning and subsequent GRPO-based reinforcement learning. The approach relies on a large GPT-annotated CoT dataset and a vision-language reward model to align reasoning with high-quality images, augmented by an adaptive entropy loss to stabilize training. Empirical results on GenEval, DPG-Bench, and T2I-Benchmark demonstrate consistent improvements over strong baselines, establishing a principled baseline for think-and-generate multimodal content. The work also provides dataset and training code releases to accelerate future research in reasoning-guided multimodal generation.

Abstract

Although chain-of-thought reasoning and reinforcement learning (RL) have driven breakthroughs in NLP, their integration into generative vision models remains underexplored. We introduce ReasonGen-R1, a two-stage framework that first imbues an autoregressive image generator with explicit text-based "thinking" skills via supervised fine-tuning on a newly generated reasoning dataset of written rationales, and then refines its outputs using Group Relative Policy Optimization. To enable the model to reason through text before generating images, We automatically generate and release a corpus of model crafted rationales paired with visual prompts, enabling controlled planning of object layouts, styles, and scene compositions. Our GRPO algorithm uses reward signals from a pretrained vision language model to assess overall visual quality, optimizing the policy in each update. Evaluations on GenEval, DPG, and the T2I benchmark demonstrate that ReasonGen-R1 consistently outperforms strong baselines and prior state-of-the-art models. More: aka.ms/reasongen.

Figures (5)

• Figure 1: Overall framework of ReasonGen-R1. We propose the first reinforcement learning post-training framework that enables autoregressive image generation models to output both a chain-of-thought reasoning process and the final image.
• Figure 2: Left: We show side-by-side visualizations of images generated by Janus-Pro-7B and ReasonGen-R1 using identical prompts (prompts are summarized; see the raw prompts in Table \ref{['tab:geneval']}, \ref{['tab:dpg']}). Right: we present a performance comparison across three instruction-following benchmarks. In every benchmark, ReasonGen-R1 outperforms the base Janus-Pro-7B model, demonstrating a substantial improvement in its ability to follow instructions.
• Figure 3: The pipeline for reinforcement learning in ReasonGen-R1.
• Figure 4: Comparison on Entropy Loss regularization.
• Figure 5: Ranked word‐frequency distribution across 1000 Chain-of-Thought (CoT) rollouts. Only words appearing in at least 20% of CoTs are shown, and the three most common function words (“a”, “an”, and “and”) have been removed to highlight more informative terms.