A-IPO: Adaptive Intent-driven Preference Optimization

TL;DR

This work tackles the challenge of aligning LLMs to diverse, pluralistic user intents beyond majority viewpoints. It proposes Adaptive Intent-driven Preference Optimization (A-IPO), which augments RLHF-based preference learning with an explicit intention module and an intention–response similarity term to increase the log-odds margin by $λ\,Δ\mathrm{sim}$, improving both alignment and robustness. The authors provide theoretical guarantees showing margin and likelihood improvements when conditioning on latent intent and demonstrate substantial empirical gains across real-world and adversarial benchmarks, including Real-Pref, Attack-Pref, and GlobalOpinionQA-Ext, with metrics like Win Rate, ICS, RIC, RS, and DSR. Empirically, A-IPO shows strong performance gains, especially on minority and culturally nuanced prompts, and exhibits enhanced resilience to adversarial prompts, indicating practical impact for pluralistic and safety-conscious deployment of LLMs.

Abstract

Human preferences are diverse and dynamic, shaped by regional, cultural, and social factors. Existing alignment methods like Direct Preference Optimization (DPO) and its variants often default to majority views, overlooking minority opinions and failing to capture latent user intentions in prompts. To address these limitations, we introduce \underline{\textbf{A}}daptive \textbf{\underline{I}}ntent-driven \textbf{\underline{P}}reference \textbf{\underline{O}}ptimization (\textbf{A-IPO}). Specifically,A-IPO introduces an intention module that infers the latent intent behind each user prompt and explicitly incorporates this inferred intent into the reward function, encouraging stronger alignment between the preferred model's responses and the user's underlying intentions. We demonstrate, both theoretically and empirically, that incorporating an intention--response similarity term increases the preference margin (by a positive shift of $λ\,Δ\mathrm{sim}$ in the log-odds), resulting in clearer separation between preferred and dispreferred responses compared to DPO. For evaluation, we introduce two new benchmarks, Real-pref, Attack-pref along with an extended version of an existing dataset, GlobalOpinionQA-Ext, to assess real-world and adversarial preference alignment. Through explicit modeling of diverse user intents,A-IPO facilitates pluralistic preference optimization while simultaneously enhancing adversarial robustness in preference alignment. Comprehensive empirical evaluation demonstrates that A-IPO consistently surpasses existing baselines, yielding substantial improvements across key metrics: up to +24.8 win-rate and +45.6 Response-Intention Consistency on Real-pref; up to +38.6 Response Similarity and +52.2 Defense Success Rate on Attack-pref; and up to +54.6 Intention Consistency Score on GlobalOpinionQA-Ext.

Figures (3)

• Figure 1: Workflow of the proposed framework (A-IPO).
• Figure 2: GPT-2 Large
• Figure 3: Pythia-2.8B

Theorems & Definitions (22)

• Theorem 5.1: Extension of Theorem 1 of DPO rafailov2023direct
• proof
• Lemma 5.1: Feature augmentation reduces Bayes risk
• Theorem 5.2: Likelihood improvement under conditioning
• Lemma 5.2: Margin shift
• Theorem 5.3: NLL improvement
• Lemma D.1: Sufficient condition for intent-aligned preference
• proof
• Corollary D.1: Target margin
• Corollary D.2: Target preference level