NextQuill: Causal Preference Modeling for Enhancing LLM Personalization

Abstract

Personalizing large language models (LLMs) for individual users has become increasingly important as they are progressively integrated into real-world applications to support users' daily lives. However, existing personalization approaches often fail to distinguish which components of model predictions and training data truly reflect user preferences, leading to superficial personalization alignment. In this paper, we introduce NextQuill, a novel LLM personalization alignment framework grounded in causal preference modeling. We approach personalization from a causal perspective, treating both model predictions and ground-truth data generation as outcomes influenced by user preferences, along with other factors. We define the true preference effect as the causal impact of user history (which reflects preferences) on each token prediction or data generation instance, estimated through causal intervention techniques. Building on this insight, NextQuill introduces two complementary alignment strategies: (1) aligning model-internal causal preference effects on predictions with those reflected in ground-truth data, rather than indiscriminately fitting predictions, and (2) focusing on fitting preference-bearing tokens identified via ground-truth data preference effects, rather than treating all tokens uniformly. By integrating these strategies, NextQuill shifts the alignment process toward learning from causal preference effects, facilitating more effective and personalized adaptation. Experiments across multiple personalization benchmarks demonstrate that NextQuill significantly improves personalization quality, offering a principled, causal foundation for LLM personalization. Our codes are available on https://github.com/juntaoyou/NextQuill.

Figures (12)

• Figure 1: Illustration of NextQuill. The model-side causal graph depicts the response prediction process of LLMs, in which the outcome is influenced by user history, context information, and their interaction. The data-side causal graph describes how users generate the ground-truth response, which is jointly influenced by user characteristics, the context, and their interaction. Based on these, NextQuill introduces the causal preference modeling, defining causal preference effects on both model and data sides, and accordingly introduces two causal preference alignment strategies (i.e., two losses) to enhance the personalization of LLMs.
• Figure 2: Effect of hyper-parameter $\alpha$ on the performance.
• Figure 3: Logit difference distributions with and without user history across three benchmark datasets.
• Figure 4: Causal graphs and corresponding preference effects. The left panel shows the model-side causal graph, and the right panel shows the data-side causal graph.
• Figure 5: Visualization of token clustering based on hidden representations on three datasets.