UniRL-Zero: Reinforcement Learning on Unified Models with Joint Language Model and Diffusion Model Experts

TL;DR

UniRL-Zero addresses the gap in reinforcement learning for unified models that jointly reason with language-models and generate with diffusion-models. It defines six RL scenarios, builds a simple base unified model with a frozen LM and a diffusion DM, and optimizes a joint policy via Group Relative Policy Optimization to coordinate discrete LM actions and continuous DM denoising steps. Empirically, it demonstrates competitive base-model performance on GenEval and shows RL-driven gains across text-to-image generation, CoT-enhanced generation, instructional editing, and image-reflection tasks, using a transparent, reproducible pipeline. The work establishes a foundation for scalable, end-to-end RL in unified LM–DM systems, while acknowledging reward bias and experimental scale as important directions for future improvement.

Abstract

We present UniRL-Zero, a unified reinforcement learning (RL) framework that boosts, multimodal language model understanding and reasoning, diffusion model multimedia generation, and their beneficial interaction capabilities within a unified model. Our work defines six scenarios for unified model reinforcement learning, providing systematic baselines for reinforcement learning of unified understanding and generation model. Our code is available at https://github.com/G-U-N/UniRL.

Figures (9)

• Figure 1: Overview of the UniRL-Zero framework.
• Figure 2: Visual examples of image changes during RL training under JPEG compressibility and incompressibility rewards.
• Figure 3: Training curves of the CoT-enhanced text-to-image generation.
• Figure 4: Generation examples of the CoT-enhanced text-to-image generation. Original prompts are on the left of images and improved prompts are above images.
• Figure 5: Visual examples showcasing the effectiveness of Cycle Edit RL, highlighting enhanced similarity to the reference image and improved instrunction following.