HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation

TL;DR

HyPerAlign tackles the problem of user-specific LLM alignment by introducing a two-stage, hypothesis-driven personalization framework. It first infers user attributes and data-driven hypotheses (HypoGenic) from few-shot demonstrations, then steers in-context generations by prompting with these hypotheses or persona descriptions, without fine-tuning (cap of $10$ hypotheses). Empirical results on authorship attribution and deliberative alignment show HyPerAlign yields high win-rates against strong baselines (often >$90\%$) and dramatically reduces harmfulness (up to $70\%$ on jailbreaking benchmarks), with hypotheses that generalize across models and datasets. The work provides an interpretable, sample-efficient path to personalized, context-aware AI assistants, while noting limitations related to prompt sensitivity, context length, and potential hallucinations.

Abstract

Alignment algorithms are widely used to align large language models (LLMs) to human users based on preference annotations. Typically these (often divergent) preferences are aggregated over a diverse set of users, resulting in fine-tuned models that are aligned to the ``average-user'' preference. Nevertheless, current models are used by individual users in very specific contexts and situations, emphasizing the need for user-dependent preference control. In this work we address the problem of personalizing LLM outputs to their users. We aim to generate customized responses tailored to specific individuals instead of generic outputs that emulate the collective voices of diverse populations. We propose HyPerAlign, an interpretable and sample-efficient hypothesis-driven personalization approach for LLM models. Given few-shot examples written by a particular user, we first infer hypotheses about their communication strategies, personality, and writing style, then prompt LLM models with these hypotheses and user-specific attributes to generate customized outputs. We conduct experiments on two different personalization tasks, namely authorship attribution and deliberative alignment, with datasets from diverse domains (news articles, blog posts, emails, jailbreaking benchmarks). Results demonstrate the superiority of hypothesis-driven LLM personalization compared to preference-based fine-tuning methods. For authorship attribution, HyPerAlign generations have consistently high win-rates (commonly $> 90\%$) against state-of-the-art preference fine-tuning approaches across diverse user profiles and LLM models. For deliberative alignment, the helpfulness of LLM models is improved by up to $70\%$ on average. Overall, HyPerAlign represents an interpretable and sample-efficient strategy for the personalization of LLM models to individual users.

Figures (1)

• Figure 1: Interpretable personalized LLM alignment via hypothesis generation. Given a few representative user demonstrations, HyPerAlign learns hypotheses about the user which are then used for the in-context personalized alignment of LLM models to individual users.