ConsistentRFT: Reducing Visual Hallucinations in Flow-based Reinforcement Fine-Tuning

TL;DR

This work analyzes visual hallucinations that arise when refining flow-based diffusion models with reinforcement signals. It identifies limited exploration and trajectory imitation as root causes and proposes ConsistentRFT, a framework combining Dynamic Granularity Rollout (DGR) and Consistent Policy Gradient Optimization (CPGO) to balance global semantics with local details and to preserve cross-step velocity consistency. A Visual Hallucination Evaluator (VH-Evaluator) couples low-level artifact metrics with high-level multimodal judgments to quantify hallucinations. Empirical results demonstrate substantial reductions in hallucinations (up to 49% for low-level and 38% for high-level) and improved out-of-domain performance (+5.1%) on FLUX1.dev, outperforming SoTA baselines and demonstrating compatibility with online DPO, DDPO, and GRPO variants. The method offers a practical path to more reliable, semantics-consistent image generation under reinforcement fine-tuning in flow-based models.

Abstract

Reinforcement Fine-Tuning (RFT) on flow-based models is crucial for preference alignment. However, they often introduce visual hallucinations like over-optimized details and semantic misalignment. This work preliminarily explores why visual hallucinations arise and how to reduce them. We first investigate RFT methods from a unified perspective, and reveal the core problems stemming from two aspects, exploration and exploitation: (1) limited exploration during stochastic differential equation (SDE) rollouts, leading to an over-emphasis on local details at the expense of global semantics, and (2) trajectory imitation process inherent in policy gradient methods, distorting the model's foundational vector field and its cross-step consistency. Building on this, we propose ConsistentRFT, a general framework to mitigate these hallucinations. Specifically, we design a Dynamic Granularity Rollout (DGR) mechanism to balance exploration between global semantics and local details by dynamically scheduling different noise sources. We then introduce a Consistent Policy Gradient Optimization (CPGO) that preserves the model's consistency by aligning the current policy with a more stable prior. Extensive experiments demonstrate that ConsistentRFT significantly mitigates visual hallucinations, achieving average reductions of 49\% for low-level and 38\% for high-level perceptual hallucinations. Furthermore, ConsistentRFT outperforms other RFT methods on out-of-domain metrics, showing an improvement of 5.1\% (v.s. the baseline's decrease of -0.4\%) over FLUX1.dev. This is \href{https://xiaofeng-tan.github.io/projects/ConsistentRFT}{Project Page}.

Figures (17)

• Figure 1: Visual hallucinations in flow-based reinforcement fine-tuning. We observe that (d) existing methods may learn a shortcut: boosting in-domain reward by injecting irrelevant details. We attribute this to a constrained exploration domain, discussed in Sec. \ref{['sec:motivation']}. See Fig. \ref{['supp:vh_large']} for clearer over-optimization details.
• Figure 2: Relationship between exploration (rollout) and exploitation (optimization). We observe that the groups exhibiting coarse/fine-grained differences obtained during the rollout stage may steer the model toward coarse/fine-grained optimization. Ideally, the model should not focus solely on fine-grained details or on coarse-grained semantics. For more quantitative analysis, please refer to App. \ref{['app:com_fine_coarse_dynamic']}.
• Figure 3: Toy example of trajectory imitation optimization in policy gradient methods.
• Figure 4: Overview of ConsistentRFT, consisting of (i) Dynamic Granularity Rollout, and (ii) Consistent Policy Gradient Optimization.
• Figure 5: Reward curves. (a) reward curve over training iterations with FlowGRPO trained using PickScore; (b) reward curve over training iterations with DanceGRPO trained using HPSv2.1.

Theorems & Definitions (6)

• Corollary 1: Reinterpretation of GRPO
• proof : Proof of Corollary 1
• proof : Proof of DDPO Corollary
• proof : Proof of DPO Corollary
• Theorem S1: Consistency Property of Flow Models
• proof : Proof of Theorem \ref{['thm:cpgo_consistency']}