MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples

TL;DR

MPPO introduces a reward-fitting framework that uses the average likelihood of model responses to align LLMs without a reference model, enabling effective preference optimization in sparse data with multiple responses per query. Among point-wise, pair-wise, and list-wise implementations, the pair-wise approach, especially Pair-MNM, delivers the strongest gains, surpassing state-of-the-art methods on MT-Bench and approaching top performance on Arena-Hard. The method demonstrates data-efficient training, robust performance across benchmarks, and practical applicability by removing the need for a separate reward or reference model. Overall, MPPO advances LLM alignment by leveraging multi-response preferences and demonstrating clear advantages in real-world, data-scarce settings.

Abstract

Aligning Large Language Models (LLMs) with human feedback is crucial for their development. Existing preference optimization methods such as DPO and KTO, while improved based on Reinforcement Learning from Human Feedback (RLHF), are inherently derived from PPO, requiring a reference model that adds GPU memory resources and relies heavily on abundant preference data. Meanwhile, current preference optimization research mainly targets single-question scenarios with two replies, neglecting optimization with multiple replies, which leads to a waste of data in the application. This study introduces the MPPO algorithm, which leverages the average likelihood of model responses to fit the reward function and maximizes the utilization of preference data. Through a comparison of Point-wise, Pair-wise, and List-wise implementations, we found that the Pair-wise approach achieves the best performance, significantly enhancing the quality of model responses. Experimental results demonstrate MPPO's outstanding performance across various benchmarks. On MT-Bench, MPPO outperforms DPO, ORPO, and SimPO. Notably, on Arena-Hard, MPPO surpasses DPO and ORPO by substantial margins. These achievements underscore the remarkable advantages of MPPO in preference optimization tasks.

Figures (4)

• Figure 1: A simple example: When answering questions, the LLMs may generate multiple responses, but the quality of different responses varies.
• Figure 2: The primary differences between MPPO and DPO are three-fold: (1) MPPO directly models the reward function using the average likelihood of the responses. (2) MPPO can utilize any number of negative samples for training in sparse scenarios. (3) The optimization objective of MPPO does not require a reference model or any hyperparameters, making the model more stable.
• Figure 3: Training Logs for Point-CE (N=3). (a) shows the variation in average likelihood for the chosen and three rejected samples; (b) illustrates the overall loss variation during the training period.
• Figure 4: Based on the Pair-wise implementation approach, the training records (N=3) are organized and listed from top to bottom as follows: (a) and (b). Pair-Single implementation; (c) and (d). Pair-MNS implementation; (e) and (f). Pair-MNM implementation; (g) and (h). Pair-MCM implementation.