Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Difference Feedback: Generating Multimodal Process-Level Supervision for VLM Reinforcement Learning

Feiding, Yongkang Zhang, Yuhao Liao, Zijian Zeng, Chunzheng Zhu, Yaozong Zheng, Yafei Liu, Yeling Peng, Youwei Wang, Sibo Wang, Huiming Yang, Linglin Liao, Shunzhi Yang

Abstract

Vision--language models (VLMs) are increasingly aligned via Group Relative Policy Optimization (GRPO)-style training. However, relying solely on terminal outcome rewards yields sparse credit assignment in multi-step reasoning, weakening the linkage between visual evidence and intermediate steps and often causing unstable optimization and visual hallucinations. We propose Differential Feedback, which automatically constructs token/step-level supervision masks by repairing erroneous reasoning trajectories, explicitly marking the key positions that require correction. Without costly large-scale step-by-step human annotations, our method enables process-level visual alignment and can be seamlessly integrated into existing GRPO-like frameworks. Experiments on multimodal reasoning benchmarks including MMMStar and MathVista show an average 3% improvement under matched compute budgets. Our approach offers an effective, low-cost solution for accurate vision--reasoning process alignment.

Difference Feedback: Generating Multimodal Process-Level Supervision for VLM Reinforcement Learning

Abstract

Vision--language models (VLMs) are increasingly aligned via Group Relative Policy Optimization (GRPO)-style training. However, relying solely on terminal outcome rewards yields sparse credit assignment in multi-step reasoning, weakening the linkage between visual evidence and intermediate steps and often causing unstable optimization and visual hallucinations. We propose Differential Feedback, which automatically constructs token/step-level supervision masks by repairing erroneous reasoning trajectories, explicitly marking the key positions that require correction. Without costly large-scale step-by-step human annotations, our method enables process-level visual alignment and can be seamlessly integrated into existing GRPO-like frameworks. Experiments on multimodal reasoning benchmarks including MMMStar and MathVista show an average 3% improvement under matched compute budgets. Our approach offers an effective, low-cost solution for accurate vision--reasoning process alignment.

Paper Structure

This paper contains 42 sections, 30 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Notation and Problem Setup
  5. Difference Feedback Generation (DF)
  6. Core idea.
  7. Train the Repair Model: Two-Stage (Supervised Fine-Tuning(SFT) + RL)
  8. Stage 1: SFT Warm Start
  9. Stage 2: Learning to Repair with Reinforcement Learning
  10. Reward-hacking auditing (gating).
  11. Image-removal penalty (C1).
  12. Final repair reward (with auditing and C1).
  13. Obtaining Difference Feedback
  14. Difference Alignment and Supervision Mask.
  15. Soft Weights (Optional).
  16. ...and 27 more sections

Figures (1)

  • Figure 1: Difference Feedback (DF) provides fine-grained process supervision for VLM alignment. When the policy produces an incorrect trajectory, a small-edit repair is generated; the difference between the two outputs yields a token-level mask that gates gradient updates.