Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimization

TL;DR

ADPO tackles the high cost of parallel test-time scaling by training a single policy to both generate answers and output calibrated self-verification scores. It introduces a preference-based verification reward to maintain informative gradients under class imbalance and an advantage decoupling mechanism with token-level masks to prevent reward hacking and gradient interference. Empirically, ADPO yields substantial gains in verification AUC and task accuracy across MathVista, MMMU, ReasonSeg, and GUI/Android benchmarks, while also reducing inference latency. The approach enables reliable best-of-N candidate selection in multimodal reasoning with lower deployment cost, making robust test-time scaling practical for vision-language models. All mathematical expressions are presented with proper delimitation, e.g., $R^p_i$ and $\mathcal{J}(\theta)$ adopt $...$ formatting where used.

Abstract

Parallel test-time scaling typically trains separate generation and verification models, incurring high training and inference costs. We propose Advantage Decoupled Preference Optimization (ADPO), a unified reinforcement learning framework that jointly learns answer generation and self-verification within a single policy. ADPO introduces two innovations: a preference verification reward improving verification capability and a decoupled optimization mechanism enabling synergistic optimization of generation and verification. Specifically, the preference verification reward computes mean verification scores from positive and negative samples as decision thresholds, providing positive feedback when prediction correctness aligns with answer correctness. Meanwhile, the advantage decoupled optimization computes separate advantages for generation and verification, applies token masks to isolate gradients, and combines masked GRPO objectives, preserving generation quality while calibrating verification scores. ADPO achieves up to +34.1% higher verification AUC and -53.5% lower inference time, with significant gains of +2.8%/+1.4% accuracy on MathVista/MMMU, +1.9 cIoU on ReasonSeg, and +1.7%/+1.0% step success rate on AndroidControl/GUI Odyssey.

Figures (11)

• Figure 1: Overview of ADPO. We build a unified reinforcement learning framework that jointly learns answer generation and self-verification within a single policy. The unified policy model provides reliable scoring that enables effective test-time scaling via best-of-N selection, significantly improving performance across multimodal tasks.
• Figure 2: Effect of class imbalance on the Binary Verification Reward. The Blue line shows the proportion of correct answers among responses with binary verification reward = 1. The Orange line shows the proportion of answers with verification score = 1.
• Figure 3: The framework of ADPO. Given a multimodal input, our unified policy produces an answer and a self-verification score to rank answer candidates. We design a preference verification reward to improve verification capability and a decoupled optimization mechanism to enable synergistic optimization of generation and verification. Preference verification reward aligns verification scores with answer correctness by providing relative ranking supervision. Advantage decoupled optimization computes separate advantages for generation and verification, and applies token masks to isolate gradients, thereby preventing reward hacking and reducing gradient interference between the two objectives.
• Figure 4: AP improvement
• Figure 5: AUC improvement