APO: Alpha-Divergence Preference Optimization
Wang Zixian
TL;DR
APO introduces an anchored, $\alpha$-divergence-based framework for online LLM RLHF that continuously interpolates between mode-covering and mode-seeking updates. By deriving a unified gradient $\nabla_\theta D_\alpha = -\frac{1}{\alpha} \mathbb{E}_{i\sim p_\theta}[r(i)^\alpha \nabla_\theta \log p_\theta(i)]$ and employing a Boltzmann soft target over group candidates, APO enables smooth transitions from $KL(q\|p)$ to $KL(p\|q)$ through $\alpha \in (0,1)$, while anchoring in log-ratio space provides a stable geometry. A reward + confidence guarded scheduling strategy dynamically lowers $\alpha$ only when the policy is confident (low entropy) and improving (positive reward gain), mitigating “confident-but-wrong” failures. Empirically, APO achieves competitive results against GRPO/GSPO baselines on Qwen3-1.7B with math-level3 and demonstrates stable training across multiple $\alpha$-scheduling variants, illustrating the practical viability of scheduling divergences within anchored RLHF. The approach offers a principled pathway to adapt teaching signals from broad coverage to targeted exploitation in complex instruction tuning tasks.
Abstract
Two divergence regimes dominate modern alignment practice. Supervised fine-tuning and many distillation-style objectives implicitly minimize the forward KL divergence KL(q || pi_theta), yielding stable mode-covering updates but often under-exploiting high-reward modes. In contrast, PPO-style online reinforcement learning from human feedback behaves closer to reverse KL divergence KL(pi_theta || q), enabling mode-seeking improvements but risking mode collapse. Recent anchored methods, such as ADPO, show that performing the projection in anchored coordinates can substantially improve stability, yet they typically commit to a single divergence. We introduce Alpha-Divergence Preference Optimization (APO), an anchored framework that uses Csiszar alpha-divergence to continuously interpolate between forward and reverse KL behavior within the same anchored geometry. We derive unified gradient dynamics parameterized by alpha, analyze gradient variance properties, and propose a practical reward-and-confidence-guarded alpha schedule that transitions from coverage to exploitation only when the policy is both improving and confidently calibrated. Experiments on Qwen3-1.7B with math-level3 demonstrate that APO achieves competitive performance with GRPO and GSPO baselines while maintaining training stability.