Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Preference Alignment via Directional Neighborhood Consensus

Ruochen Mao, Yuling Shi, Xiaodong Gu, Jiaheng Wei

TL;DR

The paper addresses the brittleness of preference-aligned LLMs when user preferences lie outside the training distribution by identifying a preference coverage gap. It introduces Robust Preference Selection (RPS), a training-free, post-hoc method that samples multiple directions in the local neighborhood of a target preference and selects the best response according to the original target, supported by a theoretical framework and a proof of superiority over single-direction baselines. The approach is validated across three alignment paradigms (DPA, DPO, SFT) and three datasets, achieving up to a 69% win rate on challenging out-of-distribution preferences, demonstrating practical, model-agnostic robustness improvements without retraining. The work provides a principled, inference-time solution to enhance reliability and controllability of AI systems in diverse user scenarios.

Abstract

Aligning large language models with human preferences is critical for creating reliable and controllable AI systems. A human preference can be visualized as a high-dimensional vector where different directions represent trade-offs between desired attributes (e.g., helpfulness vs. verbosity). Yet, because the training data often reflects dominant, average preferences, LLMs tend to perform well on common requests but fall short in specific, individual needs. This mismatch creates a preference coverage gap. Existing methods often address this through costly retraining, which may not be generalized to the full spectrum of diverse preferences. This brittleness means that when a user's request reflects a nuanced preference deviating from the training data's central tendency, model performance can degrade unpredictably. To address this challenge, we introduce Robust Preference Selection (RPS), a post-hoc, training-free method by leveraging directional neighborhood consensus. Instead of forcing a model to generate a response from a single, highly specific preference, RPS samples multiple responses from a local neighborhood of related preferences to create a superior candidate pool. It then selects the response that best aligns with the user's original intent. We provide a theoretical framework showing our neighborhood generation strategy is provably superior to a strong baseline that also samples multiple candidates. Comprehensive experiments across three distinct alignment paradigms (DPA, DPO, and SFT) demonstrate that RPS consistently improves robustness against this baseline, achieving win rates of up to 69% on challenging preferences from under-represented regions of the space without any model retraining. Our work presents a practical, theoretically-grounded solution for enhancing the reliability of preference-aligned models.

Robust Preference Alignment via Directional Neighborhood Consensus

TL;DR

The paper addresses the brittleness of preference-aligned LLMs when user preferences lie outside the training distribution by identifying a preference coverage gap. It introduces Robust Preference Selection (RPS), a training-free, post-hoc method that samples multiple directions in the local neighborhood of a target preference and selects the best response according to the original target, supported by a theoretical framework and a proof of superiority over single-direction baselines. The approach is validated across three alignment paradigms (DPA, DPO, SFT) and three datasets, achieving up to a 69% win rate on challenging out-of-distribution preferences, demonstrating practical, model-agnostic robustness improvements without retraining. The work provides a principled, inference-time solution to enhance reliability and controllability of AI systems in diverse user scenarios.

Abstract

Aligning large language models with human preferences is critical for creating reliable and controllable AI systems. A human preference can be visualized as a high-dimensional vector where different directions represent trade-offs between desired attributes (e.g., helpfulness vs. verbosity). Yet, because the training data often reflects dominant, average preferences, LLMs tend to perform well on common requests but fall short in specific, individual needs. This mismatch creates a preference coverage gap. Existing methods often address this through costly retraining, which may not be generalized to the full spectrum of diverse preferences. This brittleness means that when a user's request reflects a nuanced preference deviating from the training data's central tendency, model performance can degrade unpredictably. To address this challenge, we introduce Robust Preference Selection (RPS), a post-hoc, training-free method by leveraging directional neighborhood consensus. Instead of forcing a model to generate a response from a single, highly specific preference, RPS samples multiple responses from a local neighborhood of related preferences to create a superior candidate pool. It then selects the response that best aligns with the user's original intent. We provide a theoretical framework showing our neighborhood generation strategy is provably superior to a strong baseline that also samples multiple candidates. Comprehensive experiments across three distinct alignment paradigms (DPA, DPO, and SFT) demonstrate that RPS consistently improves robustness against this baseline, achieving win rates of up to 69% on challenging preferences from under-represented regions of the space without any model retraining. Our work presents a practical, theoretically-grounded solution for enhancing the reliability of preference-aligned models.
Paper Structure (27 sections, 1 theorem, 3 equations, 6 figures, 6 tables, 1 algorithm)

This paper contains 27 sections, 1 theorem, 3 equations, 6 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preference Alignment in Large Language Models
  4. Enhancing Robustness in Language Models
  5. Problem Setup and Theoretical Framework
  6. Preference Space and Reward Model
  7. The Preference Coverage Problem
  8. Neighborhood Consensus Theory
  9. Robust Preference Selection Algorithm
  10. Experiments
  11. Experimental Setup
  12. Models and Datasets
  13. Evaluation Protocol
  14. Results
  15. RPS Consistently Outperforms the Baseline and Excels on OOD Preferences
  16. ...and 12 more sections

Key Result

Theorem 1

Let $S_{\text{RPS}} = \{s_1, \ldots, s_k\}$ be the set of scores from $k$ responses generated from the neighborhood $\mathcal{N}_k$, where $s_i = \mathbf{v}_{\text{target}}^T \mathbf{r}(x, y_i)$. Let $S_{\text{Baseline}} = \{s'_1, \ldots, s'_k\}$ be the set of scores from $k$ responses generated fro This performance gap, illustrated in Figure fig:theorem_illustration, represents the robustness gai

Figures (6)

  • Figure 1: Illustration of the preference coverage gap. While user preferences (a) can span the entire space, the model's training data (b) is often concentrated on dominant, average preferences, leaving individual needs in a sparse subset. This creates a gap where a user's target preference may lie.
  • Figure 2: Conceptual visualization of RPS. Instead of relying on a single, potentially out-of-distribution target preference $\mathbf{v}_{\text{target}}$ (solid black arrow), RPS samples $k$ directions from its local neighborhood (dashed blue arrows). By generating responses from this diverse set, RPS can identify a response that better aligns with the user's true intent.
  • Figure 3: Conceptual illustration of Theorem 1. The distributions represent the scores of candidate responses for the baseline (orange) and RPS (blue). Under Assumption 1, the RPS candidate pool is drawn from a higher-quality distribution. Consequently, the expected score of the best RPS response, $\mathbb{E}[s^*_{\text{RPS}}]$, is strictly greater than that of the best baseline response, $\mathbb{E}[s^*_{\text{baseline}}]$. This difference is the robustness gain.
  • Figure 4: Overall RPS win rates by model (DPA, DPO, SFT) and dataset. Bars show mean win rates across all tested preference directions.
  • Figure 5: Directional robustness. RPS win rate vs. preference angle for DPA (left), DPO (middle), and SFT (right) models. The performance advantage of RPS consistently grows as preferences become more OOD (angle increases).
  • ...and 1 more figures

Theorems & Definitions (2)

  • Theorem 1: Superiority of Neighborhood Generation
  • proof