Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policy

TL;DR

This work tackles perception–reasoning decoupling in multimodal RLVR by introducing Thinking with Deltas and the Differential Visual Reasoning Policy (DVRP). DVRP leverages visual triplets—Invariant, Decremental, and Incremental views—to provide intrinsic supervision that enforces visual sensitivity to occlusion and visual robustness to perturbations, while maintaining reward-driven learning from GRPO. Across mathematical and medical benchmarks, DVRP yields state-of-the-art results on 3B and 7B backbones and demonstrates strong generalization to domain foundation models, all without external annotations or tools. The findings show that explicitly coupling visual input changes to reasoning paths produces robust, visually grounded multimodal reasoning with practical implications for efficient, scalable AI systems.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced reasoning capabilities in Large Language Models. However, adapting RLVR to multimodal domains suffers from a critical \textit{perception-reasoning decoupling}. Existing paradigms, driven by text-centric outcome rewards, reasoning in language medium, inadvertently encourage models to bypass visual perception. We empirically validate this through blind experiments: state-of-the-art policies maintain or surprisingly improve performance even when visual inputs are entirely removed. This reveals that these models degenerate into \textit{blind reasoners}, exploiting linguistic priors to generate plausible answers instead of attending to visual evidence. In response, we propose \textbf{Thinking with Deltas}, a framework driven by a \textbf{Differential Visual Reasoning Policy (DVRP)}. DVRP introduces intrinsic supervision via visual triplets, comprising original, masked, and perturbed inputs. It optimizes the model to maximize reasoning divergence from masked inputs (enforcing \textit{visual sensitivity}) while minimizing divergence from perturbed inputs (ensuring \textit{visual robustness}). By aligning reasoning variations strictly with the \textit{Delta} of visual information, DVRP inherently bolsters visual understanding capabilities and significantly outperforms state-of-the-art methods on both general and medical benchmarks, without requiring external annotations or auxiliary tools.

Figures (4)

• Figure 1: Empirical Validation of Visual Decoupling. Performance comparison in blind settings (Text Only or Blank Image) reveals latent reward hacking. The negligible performance drop in GRPO and the unexpected performance gain in DAPO (where removing visual inputs actually improves accuracy) indicate that policies degenerate into blind reasoners relying on linguistic shortcuts rather than visual evidence.
• Figure 2: The framework of DVRP. Our method bridges the perception-reasoning decoupling via a visual triplet contrastive learning objective. The upper stream represents the standard reasoning rollout. The lower streams enforce two critical visual properties: (1) Visual Robustness: minimizing the KL-divergence ($KL_{noise}$) between predictions on original and noise-perturbed inputs ($+\Delta$); and (2) Visual Sensitivity: maximizing the divergence ($KL_{mask}$) when critical visual semantics are occluded ($-\Delta$). An entropy regularization term $\mathcal{H}$ prevents distribution collapse.
• Figure 3: The standardized prompt template employed across all training and evaluation phases. To ensure consistent reasoning behaviors and facilitate automated answer extraction, we explicitly instruct the model to encapsulate its chain-of-thought within <think> tags and place the final result inside a $\backslash$boxed{} command.
• Figure 4: Visual Dependency Analysis. Divergence in reasoning paths. Current MLLMs show minimal sensitivity to masked vs. perturbed inputs (consistent in top rows), but become confused when visual semantics are masked (divergent in bottom row), indicating a lack of visual grounding.