Multi-Response Preference Optimization with Augmented Ranking Dataset

TL;DR

The paper tackles the data bottleneck in preference optimization for large language models by introducing an automated dataset augmentation strategy and a multi-response learning framework. It details a four-stage augmentation pipeline that uses seed preferences to generate a large ranked dataset with a reward model to score candidates, and it introduces Multi-DPO to leverage multiple responses per prompt via a weighted objective. Empirical results show that dataset augmentation substantially boosts performance and that Multi-DPO consistently outperforms standard DPO, especially when data are scarce. The work highlights the practical value of combining automated data generation with multi-response learning for scalable preference alignment of LLMs, while noting limitations related to reward-model reliance and approximate optimization.

Abstract

Recent advancements in Large Language Models (LLMs) have been remarkable, with new models consistently surpassing their predecessors. These advancements are underpinned by extensive research on various training mechanisms. Among these, Preference Optimization has played a significant role in improving the performance of LLMs by incorporating human preferences into the training process. However, constructing preference optimization datasets is challenging and the optimization process is highly sensitive to the dataset quality. In this study, we propose a novel approach to augment Preference Optimization datasets. Additionally, we introduce a Multi-response-based Preference Optimization training method that enables the simultaneous learning of multiple responses.

Figures (2)

• Figure 1: The preference dataset augmentation is conducted through four stages. In each stage, all data is generated, trained, and evaluated by the model without human intervention.
• Figure 2: Results of MT-bench test.

Theorems & Definitions (2)

• Lemma 1
• Proof 1