ACPO: Adaptive Curriculum Policy Optimization for Aligning Vision-Language Models in Complex Reasoning

TL;DR

This work tackles the instability and data-inefficiency of RLHF for aligning large vision-language models on complex reasoning tasks. It introduces Adaptive Curriculum Policy Optimization (ACPO), which combines a dynamic curriculum that transitions from on-policy exploration to off-policy exploitation with an Advantage-Aware Adaptive Clipping (AAAC) mechanism that modulates per-sample updates by normalized token advantages, replacing a fixed PPO clipping bound. The method employs a strategic gating sampling scheme and a progressively increasing reuse count K(t) to guide the on-policy to off-policy transition, and per-sample clipping bounds defined by epsilon_high(Ã) to allocate gradient budgets where signals are strongest. Experiments on MathVista, LogicVista, MMMU-Pro, and related benchmarks show ACPO achieving state-of-the-art performance with faster convergence and improved stability versus DAPO and PAPO, demonstrating robust alignment of VLMs for complex multimodal reasoning. The approach provides a scalable, adaptive optimization paradigm that enhances sample efficiency and policy robustness in challenging multimodal tasks, with practical implications for deploying capable, reliably aligned VLMs.

Abstract

Aligning large-scale vision-language models (VLMs) for complex reasoning via reinforcement learning is often hampered by the limitations of existing policy optimization algorithms, such as static training schedules and the rigid, uniform clipping mechanism in Proximal Policy Optimization (PPO). In this work, we introduce Adaptive Curriculum Policy Optimization (ACPO), a novel framework that addresses these challenges through a dual-component adaptive learning strategy. First, ACPO employs a dynamic curriculum that orchestrates a principled transition from a stable, near on-policy exploration phase to an efficient, off-policy exploitation phase by progressively increasing sample reuse. Second, we propose an Advantage-Aware Adaptive Clipping (AAAC) mechanism that replaces the fixed clipping hyperparameter with dynamic, sample-wise bounds modulated by the normalized advantage of each token. This allows for more granular and robust policy updates, enabling larger gradients for high-potential samples while safeguarding against destructive ones. We conduct extensive experiments on a suite of challenging multimodal reasoning benchmarks, including MathVista, LogicVista, and MMMU-Pro. Results demonstrate that ACPO consistently outperforms strong baselines such as DAPO and PAPO, achieving state-of-the-art performance, accelerated convergence, and superior training stability.

Figures (2)

• Figure 1: Overview of ACPO. Unlike GRPO, ACPO removes the KL divergence constraint. Module B introduces dynamic curriculum sampling, where the $i-th$ iteration selects moderately difficult samples based on threshold $\tau$ and $N_{max}$, which then proceed to $(i+1)-th$ iteration . Module C adds advantage-based clipping, enabling safer, more effective updates for high-advantage samples.
• Figure 2: Training Dynamics of Reward and Clip Ratio in Ablation and Baseline RL Experiments.