CPPO: Contrastive Perception for Vision Language Policy Optimization

TL;DR

CPPO tackles multimodal reasoning in vision–language models by decoupling perception from reasoning through entropy-based perception token detection and teaching perception grounding with a token-level Contrastive Perception Loss (CPL). The method augments the GRPO objective with an unsupervised CPL that uses information-preserving and information-removing perturbations, applied only to perception tokens via an entropy-based top-$k$ selection. Empirical results on ViRL39K and multiple benchmarks show CPPO consistently outperforms GRPO and prior perception-aware RL methods, with notable gains for mid-sized models and improved out-of-domain generalization. The approach is efficient, model-agnostic, and scalable, providing practical gains for finetuning vision-language policies without extra auxiliary models or supervision.

Abstract

We introduce CPPO, a Contrastive Perception Policy Optimization method for finetuning vision-language models (VLMs). While reinforcement learning (RL) has advanced reasoning in language models, extending it to multimodal reasoning requires improving both the perception and reasoning aspects. Prior works tackle this challenge mainly with explicit perception rewards, but disentangling perception tokens from reasoning tokens is difficult, requiring extra LLMs, ground-truth data, forced separation of perception from reasoning by policy model, or applying rewards indiscriminately to all output tokens. CPPO addresses this problem by detecting perception tokens via entropy shifts in the model outputs under perturbed input images. CPPO then extends the RL objective function with a Contrastive Perception Loss (CPL) that enforces consistency under information-preserving perturbations and sensitivity under information-removing ones. Experiments show that CPPO surpasses previous perception-rewarding methods, while avoiding extra models, making training more efficient and scalable.

Figures (9)

• Figure 1: CPPO vs. prior perception-rewarding methods. Prior work follows three strategies: (1) Visionary-R1 and Vision-SR1 force the policy to generate separated perception from reasoning, followed by an LLM perception reward, (2) Perception-R1 uses ground-truth CoT and an LLM as a judge to provide perception reward, and (3) PAPO applies a perception loss to all rollout tokens. In contrast, CPPO uses entropy of the output tokens to identify perception tokens and assigns a Contrastive Perception Loss (CPL) exclusively to these tokens.
• Figure 2: An overview of CPPO. For each rollout $\mathbf{o}_i$, perception tokens are identified and their probability distributions are computed under three conditions: the original image $I$ (anchor sample: $\pi_{\theta}(o_{i,t})$), an information-preserving perturbation $I^{+}$ (positive sample: $\pi_{\theta}^+(o_{i,t})$), and an information-removing perturbation $I^{-}$ (negative sample: $\pi_{\theta}^-(o_{i,t})$). Similarities $sim(\pi_{\theta}(o_{i,t}), \pi_{\theta}^+(o_{i,t}))$ and $sim(\pi_{\theta}(o_{i,t}), \pi_{\theta}^-(o_{i,t}))$ are computed and incorporated into the Contrastive Perception Loss (CPL), which serves as an additional perception-specific term in the RL objective. Notations are simplified for brevity.
• Figure 3: Sample outputs generated with CPPO with top 40% detected perception tokens.
• Figure 4: CPPO vs. GRPO (avg@8) on Qwen2.5-VL-3B across in-domain and out-of-domain scenarios. The X-axis represents RL training steps. The shaded area corresponds to one standard deviation over 8 responses. 1st column: Reward comparison on the in-domain dataset during training. 2nd and 3rd columns: Comparison on four out-of-domain visual reasoning benchmarks.
• Figure 5: Sample information-removing perturbations and information-preserving perturbations.

Theorems & Definitions (2)

• Proposition 1: Entropy increase as a proxy for perception dependence
• proof