AMaPO: Adaptive Margin-attached Preference Optimization for Language Model Alignment
Ruibo Deng, Duanyu Feng, Wenqiang Lei
TL;DR
AMaPO proposes an instance-wise adaptive margin to resolve the overfitting-underfitting dilemma in offline preference optimization for LLM alignment. By estimating a per-instance margin $\gamma(x,y_w,y_l)$ using batch statistics and applying exponential scaling, AMaPO reallocates learning effort toward misranked samples while suppressing easy cases. The approach yields state-of-the-art ranking accuracy and improved downstream alignment across multiple models and domains, while showing robust generalization under ID and OOD conditions. The work provides a unified, margin-centric view of DPO-style methods and offers practical guidance for scalable, stable offline preference optimization.
Abstract
Offline preference optimization offers a simpler and more stable alternative to RLHF for aligning language models. However, their effectiveness is critically dependent on ranking accuracy, a metric where further gains are highly impactful. This limitation arises from a fundamental problem that we identify and formalize as the Overfitting-Underfitting Dilemma: current margin designs cause models to apply excessive, wasteful gradients to correctly ranked samples (overfitting) while providing insufficient corrective signals for misranked ones (underfitting). To resolve this dilemma, we propose Adaptive Margin-attached Preference Optimization (AMaPO), a simple yet principled algorithm. AMaPO employs an instance-wise adaptive margin, refined by Z-normalization and exponential scaling, which dynamically reallocates learning effort by amplifying gradients for misranked samples and suppressing them for correct ones. Extensive experiments on widely used benchmarks demonstrate that AMaPO not only achieves better ranking accuracy and superior downstream alignment performance, but targeted analysis also confirms that it successfully mitigates the core overfitting and underfitting issues.