Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment

TL;DR

This work investigates why DPO-based multi-objective alignment (MOA) struggles due to pervasive preference conflicts in training data, which impede achieving a high-quality Pareto Front. It introduces Self-Improvement DPO Towards Pareto Optimality (SIPO), a framework that automatically generates, refines, and filters Pareto-optimal responses, then uses them to fine-tune policy LLMs in a non-conflicting manner. Through extensive experiments on HelpSteer and BeaverTails, SIPO consistently yields superior Pareto Fronts and demonstrates the value of components like refinement and filtering, as well as its compatibility with existing DPO-based MOA methods. The approach reduces manual labeling needs, scales with multiple objectives, and offers a practical path to more robust MOA in real-world settings, while acknowledging computational overhead and opportunities for efficiency improvements.

Abstract

Multi-Objective Alignment (MOA) aims to align LLMs' responses with multiple human preference objectives, with Direct Preference Optimization (DPO) emerging as a prominent approach. However, we find that DPO-based MOA approaches suffer from widespread preference conflicts in the data, where different objectives favor different responses. This results in conflicting optimization directions, hindering the optimization on the Pareto Front. To address this, we propose to construct Pareto-optimal responses to resolve preference conflicts. To efficiently obtain and utilize such responses, we propose a self-improving DPO framework that enables LLMs to self-generate and select Pareto-optimal responses for self-supervised preference alignment. Extensive experiments on two datasets demonstrate the superior Pareto Front achieved by our framework compared to various baselines. Code is available at https://github.com/zyttt-coder/SIPO.

Figures (13)

• Figure 1: Illustration on the impact of preference conflicts to MOA, and how Pareto-optimal responses can mitigate such issue for superior Pareto Front.
• Figure 2: The impact of preference conflicts on Pareto Front optimization. Experiments are conducted on DPO soups with Alpaca-7B.
• Figure 3: The impact of preference conflicts on Pareto Front optimization. Experiments are conducted on MODPO with Alpaca-7B.
• Figure 4: The impact of preference conflicts on Pareto Front optimization. Results of BeaverTails with DPO soups on LLaMA-2-7B.
• Figure 5: Illustration of our proposed SIPO framework.