Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VAR RL Done Right: Tackling Asynchronous Policy Conflicts in Visual Autoregressive Generation

Shikun Sun, Liao Qu, Huichao Zhang, Yiheng Liu, Yangyang Song, Xian Li, Xu Wang, Yi Jiang, Daniel K. Du, Xinglong Wu, Jia Jia

TL;DR

The paper addresses the instability of reinforcement learning in Visual AutoRegressive (VAR) generation caused by asynchronous policy conflicts across multi-scale token grids. It introduces a principled framework to enhance Group Relative Policy Optimization (GRPO) using Value as Middle Return (VMR), Per-Action Normalization Weighting (PANW), and Mask Propagation (MP), preserving policy optimality while stabilizing credit assignment. The authors provide theoretical results showing two-stage invariance within the VAR policy family and demonstrate substantial empirical gains on text rendering and human-preference metrics, outperforming vanilla GRPO and diffusion-based baselines. This work enables robust RL alignment for VAR models, yielding higher sample quality and better objective alignment in high-resolution text-to-image generation. Overall, the method advances reliable, scalable RL for multi-scale, parallel token generation in visual synthesis.

Abstract

Visual generation is dominated by three paradigms: AutoRegressive (AR), diffusion, and Visual AutoRegressive (VAR) models. Unlike AR and diffusion, VARs operate on heterogeneous input structures across their generation steps, which creates severe asynchronous policy conflicts. This issue becomes particularly acute in reinforcement learning (RL) scenarios, leading to unstable training and suboptimal alignment. To resolve this, we propose a novel framework to enhance Group Relative Policy Optimization (GRPO) by explicitly managing these conflicts. Our method integrates three synergistic components: 1) a stabilizing intermediate reward to guide early-stage generation; 2) a dynamic time-step reweighting scheme for precise credit assignment; and 3) a novel mask propagation algorithm, derived from principles of Reward Feedback Learning (ReFL), designed to isolate optimization effects both spatially and temporally. Our approach demonstrates significant improvements in sample quality and objective alignment over the vanilla GRPO baseline, enabling robust and effective optimization for VAR models.

VAR RL Done Right: Tackling Asynchronous Policy Conflicts in Visual Autoregressive Generation

TL;DR

The paper addresses the instability of reinforcement learning in Visual AutoRegressive (VAR) generation caused by asynchronous policy conflicts across multi-scale token grids. It introduces a principled framework to enhance Group Relative Policy Optimization (GRPO) using Value as Middle Return (VMR), Per-Action Normalization Weighting (PANW), and Mask Propagation (MP), preserving policy optimality while stabilizing credit assignment. The authors provide theoretical results showing two-stage invariance within the VAR policy family and demonstrate substantial empirical gains on text rendering and human-preference metrics, outperforming vanilla GRPO and diffusion-based baselines. This work enables robust RL alignment for VAR models, yielding higher sample quality and better objective alignment in high-resolution text-to-image generation. Overall, the method advances reliable, scalable RL for multi-scale, parallel token generation in visual synthesis.

Abstract

Visual generation is dominated by three paradigms: AutoRegressive (AR), diffusion, and Visual AutoRegressive (VAR) models. Unlike AR and diffusion, VARs operate on heterogeneous input structures across their generation steps, which creates severe asynchronous policy conflicts. This issue becomes particularly acute in reinforcement learning (RL) scenarios, leading to unstable training and suboptimal alignment. To resolve this, we propose a novel framework to enhance Group Relative Policy Optimization (GRPO) by explicitly managing these conflicts. Our method integrates three synergistic components: 1) a stabilizing intermediate reward to guide early-stage generation; 2) a dynamic time-step reweighting scheme for precise credit assignment; and 3) a novel mask propagation algorithm, derived from principles of Reward Feedback Learning (ReFL), designed to isolate optimization effects both spatially and temporally. Our approach demonstrates significant improvements in sample quality and objective alignment over the vanilla GRPO baseline, enabling robust and effective optimization for VAR models.
Paper Structure (36 sections, 4 theorems, 46 equations, 10 figures, 6 tables)

This paper contains 36 sections, 4 theorems, 46 equations, 10 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Autoregressive Image Generation
  4. Reinforcement Learning for Image Generation
  5. Methodology
  6. Preliminary
  7. Visual AutoRegressive Models
  8. VAR Sequences as MDPs
  9. The Optimal Solution for KL-Regularized RL
  10. Motivation
  11. Value as Middle Return
  12. Formulation
  13. Analysis
  14. Per-Action Normalization Weighting
  15. Mask Propagation
  16. ...and 21 more sections

Key Result

Theorem 1

At each state $\mathbf{s}_t$, the constrained optimum satisfies where $\pi^\ast$ is the global soft-optimal policy.

Figures (10)

  • Figure 1: The number of query tokens across different timesteps in VAR generation fluctuates significantly, leading to varying task similarities and potential policy conflicts during RL optimization.
  • Figure 2: Comparison of training curves between vanilla GRPO and GRPO with VMR across varying prefix scales.
  • Figure 3: The pipeline of our two-stage GRPO is as follows: we use Monte Carlo estimation to compute the reward for $\pi^\theta_{1:m-1}$, and then apply GRPO to the two states separately.
  • Figure 4: Our token masks are propagated in reverse through the model’s multi-scale hierarchy, moving from finer to coarser feature scales.
  • Figure 5: Visual Samples of text rendering task before (left) and after (right) RL optimization. The text required for each pairs: (1) Six illuminated letters ('A', 'B', 'C', 'N', 'O', 'Y') (2) "Wouldn't you rather... VOTE. By Mail." (3) "M-BOX 2.0 by: Jimmy-Fan" (4) "DESCUBRE LO QUE ESCONDEN NUESTRO CAMPING" and "UPBAN KIDS" (5) "L'ARANCAEY FIMMNA," (6) "FISHING CHALLENGE" and "GOTCHA!" (7) "Apps Way Crisps.", "Apps Wheat Crisps.", "Apps Potato Crisps." and "Apps Cheese Balls." (8) "VAR RL Done Right". Full captions are provided in the Appendix. The RL-refined outputs demonstrate improvements in correcting character misordering, erroneous glyphs, missing or extraneous characters. Better zoom in for details.
  • ...and 5 more figures

Theorems & Definitions (9)

  • Definition 1: Middle-step soft value
  • Definition 2: VAR family $\mathcal{M}_{\pi}$
  • Definition 3: Constrained optimal policy
  • Theorem 1: Reverse-KL characterization of (π^†)
  • Theorem 2: Two-stage invariance
  • Theorem 2: Reverse-KL characterization of (π^†)
  • proof
  • Theorem 2: Two-stage invariance
  • proof