Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning

TL;DR

This work tackles the critic bottleneck in RL4LLMs by reintroducing lightweight, diverse mini-critics that guide a large actor. It introduces Asymmetric Proximal Policy Optimization (AsyPPO), which trains two mini-critics on disjoint prompt shards to provide calibrated value estimates, and leverages inter-critic uncertainty to refine the policy through advantage masking and entropy filtering. The approach yields robust, data-efficient learning, with substantial performance gains over PPO-based baselines on open-source math benchmarks using as few as 5,000 samples, and it reduces memory and computation relative to symmetric PPO. The findings highlight the value of architectural innovations—specifically asymmetric critic design and uncertainty-aware loss shaping—as a practical path to scalable RL4LLM training.

Abstract

Most recent RL for LLMs (RL4LLM) methods avoid explicit critics, replacing them with average advantage baselines. This shift is largely pragmatic: conventional value functions are computationally expensive to train at LLM scale and often fail under sparse rewards and long reasoning horizons. We revisit this bottleneck from an architectural perspective and introduce Asymmetric Proximal Policy Optimization (AsyPPO), a simple and scalable framework that restores the critics role while remaining efficient in large-model settings. AsyPPO employs a set of lightweight mini-critics, each trained on disjoint prompt shards. This design encourages diversity while preserving calibration, reducing value-estimation bias. Beyond robust estimation, AsyPPO leverages inter-critic uncertainty to refine the policy update: (i) masking advantages in states where critics agree and gradients add little learning signal, and (ii) filtering high-divergence states from entropy regularization, suppressing spurious exploration. After training on open-source data with only 5,000 samples, AsyPPO consistently improves learning stability and performance across multiple benchmarks over strong baselines, such as GRPO, achieving performance gains of more than six percent on Qwen3-4b-Base and about three percent on Qwen3-8b-Base and Qwen3-14b-Base over classic PPO, without additional tricks. These results highlight the importance of architectural innovations for scalable, efficient algorithms.

Figures (11)

• Figure 1: (Left): Learnable critics naturally enhance policy stability through fine-grained value estimation and yield continuous gains as training progresses. Off-policy ratio=8, average@4 of 6 benchmarks, i.e., AIME 24, AIME 25, MATH-500, OlympiadBench, MinervaMath, and AMC 2023. (Right):AsyPPO restores the critic’s role in PPO while remaining lightweight and stable under LLM-scale training. The average clock time of training and the peak GPU memory usage of AsyPPO are significantly lower than those of the classic PPO, remain at the GRPO level.
• Figure 2: Visual intuition behind AsyPPO. Small yet expressive critics can guide larger actors effectively when their representations are well-initialized. (a): A single critic struggles to align its value signal, leading to uncertain policy updates. Two small critics reach consensus (“This way”) and provide robust, low-variance guidance to the actor. (b): By leveraging representational priors and critic agreement, AsyPPO significantly outperforms classic PPO in terms of performance on various policy models, i.e., Qwen3-4b-Base and Qwen3-8b-Base. The Y-axis represents the improvement for the initial policy. The score calculation is the same as that in \ref{['fig:overview']}.
• Figure 3: Left: The single mini-critic parameterized by Qwen3-0.6b-Base can effectively guide policies across model scales. Middle: There are significant differences in the guiding ability of the two ensemble critics for policies. Actors uniformly use Qwen3-8B-Base, while critics use Qwen3-0.6B-Base. Right: Our ensemble method intensifies the cognitive differences among mini-critics. The y-axis represents the standard deviation between the values calculated by the two mini-critics. We train on 5,000 questions sampled from DeepMath-103K he2025deepmath and evaluate policies on five challenging math benchmarks: AIME 2024, MATH-500, OlympiadBench, MinervaMath, and AMC 2023. For each question, we report the average of 4 generations.
• Figure 4: Our ensemble critics achieve positive estimation of the state involving key reasoning pattens. We follow gandhi2025cognitive, identify the reasoning behavior via GPT4-o hurst2024gpt, and hire trained Qwen3-0.6b-Base as mini-critics and Qwen3-8b-Base as the vanilla critic.
• Figure 5: (a): Agreement among critics implies the state’s downstream dynamics are well modeled by the policy, making these samples low-value for learning and best avoided for overfitting. (b): In the high data-reuse setting (UTD=4), masking the bottom $20\%$ (by value-std) boosts AsyPPO’s learning efficiency, yields an improvement of about 6 points. The accuracy records of the six benchmarks follow \ref{['fig:overview']} (b). (c): We evaluated two $5\%$ masking mechanisms on vanilla AsyPPO (baseline), i.e., entropy vs. value-std. The value-std masking produced the strongest learning efficiency benefit. Actors use Qwen3-4B-Base, while critics use Qwen3-0.6B-Base.