Reverse Preference Optimization for Complex Instruction Following

TL;DR

The paper addresses the challenge of aligning LLMs with multiple, potentially conflicting constraints in multi-turn instruction following. It introduces Reverse Preference Optimization (RPO), which reverses unsatisfied constraints to create perfect responses under a revised instruction, increasing the gap between chosen and rejected responses and reducing data collection burden. The authors build SysBank via role-driven self-play and evaluate on SysBench and Multi-IF across 8B and 70B models, showing consistent improvements over DPO and surpassing GPT-4o at scale. They analyze the impact of gap size, training data size, and general ability, showing RPO's data efficiency and robustness. The work suggests RPO as a practical foundation for multi-preference alignment and scalable to larger models.

Abstract

Instruction following (IF) is a critical capability for large language models (LLMs). However, handling complex instructions with multiple constraints remains challenging. Previous methods typically select preference pairs based on the number of constraints they satisfy, introducing noise where chosen examples may fail to follow some constraints and rejected examples may excel in certain respects over the chosen ones. To address the challenge of aligning with multiple preferences, we propose a simple yet effective method called Reverse Preference Optimization (RPO). It mitigates noise in preference pairs by dynamically reversing the constraints within the instruction to ensure the chosen response is perfect, alleviating the burden of extensive sampling and filtering to collect perfect responses. Besides, reversal also enlarges the gap between chosen and rejected responses, thereby clarifying the optimization direction and making it more robust to noise. We evaluate RPO on two multi-turn IF benchmarks, Sysbench and Multi-IF, demonstrating average improvements over the DPO baseline of 4.6 and 2.5 points (on Llama-3.1 8B), respectively. Moreover, RPO scales effectively across model sizes (8B to 70B parameters), with the 70B RPO model surpassing GPT-4o.

Figures (3)

• Figure 1: Noise in a multi-preference scenario. Any two responses that differ in at least one aspect belong to the above six types (case ① to ⑥). ABCDEF are six constraints that need to be followed, and the color in gray indicates this response failed to follow them. C: and R: represent the adherence of chosen and rejected responses to different constraints, respectively.
• Figure 2: Illustration of RPO and data construction. We first model various users and systems and then engage them in self-play. For responses sampled during the dialogue, we perform fine-grained evaluations to assess adherence to each constraint. Different types of response pairs are converted into noise-free preference pairs, as depicted in the lower part of the figure.
• Figure 3: Variation in reward differences between chosen and rejected response over training steps.