GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learning

TL;DR

GoT-R1 introduces reinforcement learning to extend Generation Chain-of-Thought reasoning into vision, using a unified multimodal language model and a dual-stage, multi-dimensional reward framework. The rewards jointly supervise the intermediate reasoning and final image, with semantic and spatial components enabling faithful compositional generation. Empirical results on T2I-CompBench and GenEval show state-of-the-art performance, with notable gains in complex prompts and attribute binding. The work demonstrates autonomous discovery of effective reasoning strategies for visual synthesis and discusses ethical considerations for scalable deployment.

Abstract

Visual generation models have made remarkable progress in creating realistic images from text prompts, yet struggle with complex prompts that specify multiple objects with precise spatial relationships and attributes. Effective handling of such prompts requires explicit reasoning about the semantic content and spatial layout. We present GoT-R1, a framework that applies reinforcement learning to enhance semantic-spatial reasoning in visual generation. Building upon the Generation Chain-of-Thought approach, GoT-R1 enables models to autonomously discover effective reasoning strategies beyond predefined templates through carefully designed reinforcement learning. To achieve this, we propose a dual-stage multi-dimensional reward framework that leverages MLLMs to evaluate both the reasoning process and final output, enabling effective supervision across the entire generation pipeline. The reward system assesses semantic alignment, spatial accuracy, and visual quality in a unified approach. Experimental results demonstrate significant improvements on T2I-CompBench benchmark, particularly in compositional tasks involving precise spatial relationships and attribute binding. GoT-R1 advances the state-of-the-art in image generation by successfully transferring sophisticated reasoning capabilities to the visual generation domain. To facilitate future research, we make our code and pretrained models publicly available at https://github.com/gogoduan/GoT-R1.

Figures (11)

• Figure 1: GoT-R1 enhances visual generation through reinforcement learning. This figure demonstrates the improvement from a GoT-finetuned model (left) to the RL-trained GoT-R1 model (right). The model before RL generates spatially misaligned reasoning process. The RL process enhances the model's semantic-spatial reasoning capabilities, as demonstrated by its Generation Chain-of-Thought, leading to a generated image that is more closely aligned with the prompt.
• Figure 2: The GoT-R1 framework illustrating the reinforcement learning process with Group Relative Policy Optimization (GRPO). Left: Overview of the candidate sampling and initial evaluation stage, where diverse reasoning chains (GoT) and corresponding image tokens are generated from an input prompt, with an MLLM-based reward model providing preliminary scoring. Right: Detailed illustration of how MLLM-based rewards and advantages facilitate model updates via GRPO.
• Figure 3: Overview of our MLLM-based dual-stage multi-dimensional reward framework. The diagram illustrates MLLM-based rewards assessing the intermediate GoT's semantic and spatial fidelity to the prompt, as well as the final image's alignment with both the prompt and the GoT.
• Figure 4: Prompt-Reasoning Spatial Reward $R_{spa}$ process. For robust spatial evaluation, the MLLM assesses bounding boxes rendered on an image from the GoT's textual coordinates, rather than processing the coordinates directly as text.
• Figure 5: Qualitative comparison among the base model Janus-Pro-7B, the GoT-finetuned checkpoint Janus-Pro-7B-GoT, and our GRPO-enhanced model GoT-R1-7B. Our model demonstrates superior performance on prompt alignment and image quality.