Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal

Yongxin Wang, Zhicheng Yang, Meng Cao, Mingfei Han, Haokun Lin, Yingying Zhu, Xiaojun Chang, Xiaodan Liang

TL;DR

CARE tackles the instability and credit-assignment challenges of verifiable-reward multimodal training by introducing an anchored-contrastive objective that tightly groups the best rollout with semantically close hard negatives, and a one-shot Reflection-Guided Resampling that repairs representative errors during training. The method adds minimal test-time overhead, relying on a post-training fine-tuning regime to boost learning signal from failures. Empirically, CARE achieves state-of-the-art macro accuracy on MathVista and MMMU-Pro and consistently outperforms GRPO, DAPO, and GSPO across multiple benchmarks and backbones, with notable gains on Qwen2.5-VL-7B (+4.62 macro points). The work also provides a mechanistic signature analysis of update dynamics and outlines practical future directions like step-level verification and anchor ensembles to further enhance verifiable multimodal reasoning.

Abstract

Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures. When all rollouts are wrong, gradients stall; when one happens to be correct, the update usually ignores why the others are close-but-wrong, and credit can be misassigned to spurious chains. We present CARE (Contrastive Anchored REflection), a failure-centric post-training framework for multimodal reasoning that turns errors into supervision. CARE combines: (i) an anchored-contrastive objective that forms a compact subgroup around the best rollout and a set of semantically proximate hard negatives, performs within-subgroup z-score normalization with negative-only scaling, and includes an all-negative rescue to prevent zero-signal batches; and (ii) Reflection-Guided Resampling (RGR), a one-shot structured self-repair that rewrites a representative failure and re-scores it with the same verifier, converting near-misses into usable positives without any test-time reflection. CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures. On Qwen2.5-VL-7B, CARE lifts macro-averaged accuracy by 4.6 points over GRPO across six verifiable visual-reasoning benchmarks; with Qwen3-VL-8B it reaches competitive or state-of-the-art results on MathVista and MMMU-Pro under an identical evaluation protocol.

CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal

TL;DR

CARE tackles the instability and credit-assignment challenges of verifiable-reward multimodal training by introducing an anchored-contrastive objective that tightly groups the best rollout with semantically close hard negatives, and a one-shot Reflection-Guided Resampling that repairs representative errors during training. The method adds minimal test-time overhead, relying on a post-training fine-tuning regime to boost learning signal from failures. Empirically, CARE achieves state-of-the-art macro accuracy on MathVista and MMMU-Pro and consistently outperforms GRPO, DAPO, and GSPO across multiple benchmarks and backbones, with notable gains on Qwen2.5-VL-7B (+4.62 macro points). The work also provides a mechanistic signature analysis of update dynamics and outlines practical future directions like step-level verification and anchor ensembles to further enhance verifiable multimodal reasoning.

Abstract

Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures. When all rollouts are wrong, gradients stall; when one happens to be correct, the update usually ignores why the others are close-but-wrong, and credit can be misassigned to spurious chains. We present CARE (Contrastive Anchored REflection), a failure-centric post-training framework for multimodal reasoning that turns errors into supervision. CARE combines: (i) an anchored-contrastive objective that forms a compact subgroup around the best rollout and a set of semantically proximate hard negatives, performs within-subgroup z-score normalization with negative-only scaling, and includes an all-negative rescue to prevent zero-signal batches; and (ii) Reflection-Guided Resampling (RGR), a one-shot structured self-repair that rewrites a representative failure and re-scores it with the same verifier, converting near-misses into usable positives without any test-time reflection. CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures. On Qwen2.5-VL-7B, CARE lifts macro-averaged accuracy by 4.6 points over GRPO across six verifiable visual-reasoning benchmarks; with Qwen3-VL-8B it reaches competitive or state-of-the-art results on MathVista and MMMU-Pro under an identical evaluation protocol.
Paper Structure (47 sections, 3 theorems, 22 equations, 16 figures, 5 tables)

This paper contains 47 sections, 3 theorems, 22 equations, 16 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Reinforcement Learning with Verifiable Reward
  4. Multimodal Large Reasoning Models
  5. Multimodal Reasoning
  6. Methodology
  7. Problem Setting and Notation
  8. CARE: Anchored-Contrastive Subgroup
  9. Anchor.
  10. Hard negatives.
  11. Negative Selection
  12. Group-Normalized Advantages
  13. Within-group normalization.
  14. Negative-penalty scaling and update-size equalization.
  15. Mechanistic signature.
  16. ...and 32 more sections

Key Result

Proposition 1

Under the assumptions above, a plug-in (delta-method) approximation that replaces the random denominator by $\sqrt{\mathbb{E}[\sigma_S^2]}$ yields where

Figures (16)

  • Figure 1: Overview of Care. Given a multimodal prompt, the policy samples a fixed number of rollouts. A programmatic verifier scores the answer with rationale and supplies the reward used to update the policy. Anchored‑contrastive (bottom-left): pick the anchor as the verified‑correct rollout with the shortest rationale; form a subgroup by selecting hard negatives that fail the verifier but are closest in rationale to the anchor. Advantages are normalized within this subgroup, and negatives are down‑weighted. Reflection‑Guided Resampling (bottom-right): pair exactly one positive with one hard negative, insert a brief repair cue, and resample the negative once—on success replace the failure; otherwise keep it with a reduced penalty.
  • Figure 2: $K'$ signature.Left:$A_{\mathrm{raw}}[y^+]$ vs. $\sqrt{K'}$. Right:$-\overline{A}_{\mathrm{raw}}[y^-]$ vs. $1/\sqrt{K'}$. Markers are bucket means over groups with realized $K'\!\in\!\{2,\ldots,7\}$; vertical bars are 95% CIs; solid lines are OLS fits on the means.
  • Figure 3: We compare CARE, CARE w/o RGR, and GRPO on Qwen2.5-VL-3B. CARE consistently reaches higher reward at matched step budgets; CARE_wo_RGR delivers most of the gain, and adding RGR yields a small, consistent, budget‑neutral boost.
  • Figure 4: Reflection success vs. Macro-Average (triggered-only).No-Cue-Resample = resample the same hard negative without the repair prompt; Random-Resample = resample without a cue and not tied to the original negative. Experiments are built on Qwen2.5-VL-3B.
  • Figure 5: Left: Cosine vs Random. Cosine-selected near-miss negatives (cosine-top$K'$) learn substantially faster and converge higher than random. Right: Nearest vs Mixed vs Farthest. Comparing nearest (top-$K'$ by $d_{\cos}$), mixed, and farthest.
  • ...and 11 more figures

Theorems & Definitions (4)

  • Proposition 1: Two-level $K$-signature under small dispersion
  • proof : Sketch
  • Corollary 1: Binary rewards
  • Corollary 2: All-negative Rescue