GAPO: Robust Advantage Estimation for Real-World Code LLMs

TL;DR

Real-world code-editing RL suffers from outlier-prone, skewed reward distributions that distort GRPO-based advantages. GAPO addresses this by adaptively identifying an outlier-free region via the highest-density interval and using its median as an adaptive $Q$, replacing the group mean in advantage computation. Across nine instruction-tuned LLMs (3B–14B) and 51,844 tasks in 10 languages, GAPO consistently improves exact-match accuracy over GRPO and DAPO with minimal overhead, demonstrating robustness and stability. The work also provides a large real-world code-editing dataset and shows that HDI-based adaptation enhances learning on easy cases and specialization on hard cases, offering a practical, plug-and-play improvement for RL-based code editing.

Abstract

Reinforcement learning (RL) is widely used for post-training large language models (LLMs) in code editing, where group-relative methods like GRPO are popular for their critic-free, normalized advantage estimation. However, in real-world code-editing scenarios, reward distributions are often skewed with unpredictable outliers, leading to distorted advantage computation and increased noise. To address this issue, we propose Group Adaptive Policy Optimization (GAPO), which adaptively finds an outlier-free highest-density interval (HDI) per prompt and then uses the median of that interval as an adaptive Q to replace the group mean in advantage calculation. This adaptive Q robustly handles skewed distributions while remaining plug-and-play and efficient. We validate GAPO on nine instruction-tuned LLMs (3B-14B) using a large internal dataset of 51,844 real-world, history-aware code-editing tasks across 10 languages, demonstrating consistent improvements in exact match accuracy over GRPO and its variant DAPO. Code is publicly available.

Figures (12)

• Figure 1: Illustration of GRPO/DAPO and our proposed GAPO, where $r_i$ and $\hat{A}_{i,t}$ denote the reward and the advantage, respectively. $Q$ is defined as the median of an adaptive highest-density interval (HDI) derived from the reward distribution of each prompt during the rollout process.
• Figure 2: Language distribution statistics of the training dataset for code edit.
• Figure 3: Performance curves on the evaluation set for GRPO, as logged by W&B. The dotted curves correspond to LLMs post-trained with the original GRPO. The curves labeled with “median-div” correspond to our proposed GAPO.
• Figure 4: Performance curves on the evaluation set for DAPO, as logged by W&B. The dotted curves correspond to LLMs post-trained with the original DAPO.
• Figure 5: Clip fraction curves on the training set for GRPO, as logged by W&B.