From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment

TL;DR

This work tackles the limits of universal LLM alignment by introducing a comprehensive 90-dimensional preference space and observable personas to enable scalable user-level alignment. It presents AlignX, a dataset with over 1.3 million persona–preference examples derived from forum data, and AlignXpert, which offers two training paradigms—in-context alignment (ICA) and preference-bridged alignment (PBA). Across four benchmarks, AlignXpert achieves substantial accuracy gains over strong universal baselines and demonstrates robust adaptation to unseen preferences and limited interaction histories, with precise controllability of user-specific preferences. The approach advances practical, user-adaptive AI by linking rich psychological signals to interpretable, controllable generation, while highlighting important considerations around privacy and bias.

Abstract

Large language models (LLMs) have traditionally been aligned through one-size-fits-all approaches that assume uniform human preferences, fundamentally overlooking the diversity in user values and needs. This paper introduces a comprehensive framework for scalable personalized alignment of LLMs. We establish a systematic preference space characterizing psychological and behavioral dimensions, alongside diverse persona representations for robust preference inference in real-world scenarios. Building upon this foundation, we introduce \textsc{AlignX}, a large-scale dataset of over 1.3 million personalized preference examples, and develop two complementary alignment approaches: \textit{in-context alignment} directly conditioning on persona representations and \textit{preference-bridged alignment} modeling intermediate preference distributions. Extensive experiments demonstrate substantial improvements over existing methods, with an average 17.06\% accuracy gain across four benchmarks while exhibiting a strong adaptation capability to novel preferences, robustness to limited user data, and precise preference controllability. These results validate our approach toward user-adaptive AI systems.

Figures (4)

• Figure 1: Illustration of the AlignX data for personalized alignment, comprising a post with two candidate responses, three types of personas that capture both behavioral patterns ($\mathcal{P}_{\textsc{ugc}}$ and $\mathcal{P}_{\textsc{pair}}$) and descriptive features ($\mathcal{P}_{\textsc{demo}}$), enabling precise preference inference and facilitating preference learning (bottom right). Notably, LLMs aligned to universal values (e.g., GPT-4o) favor Response 2, opposite to the user's personalized preference for Response 1.
• Figure 2: Overview of the alignment methods.
• Figure 3: Accuracy varying with example numbers in the persona.
• Figure 4: Pearson correlation between 90 dimensions, with dimension indices corresponding to Table \ref{['tab:space']} in the appendix.