Steerable Chatbots: Personalizing LLMs with Preference-Based Activation Steering

TL;DR

This work introduces Preference-Based Activation Steering, an inference-time technique that injects steer vectors into an LLM's residual stream to bias outputs toward user-defined preference dimensions (e.g., cost, ambiance, culture, age, time). By constructing per-dimension steering vectors from contrastive exemplars and applying them with a strength factor d^u, the approach enables cold-start personalization without retraining or long user histories. The authors validate the method across five open-source LLMs and five dimensions through computational experiments, and further evaluate three steerable interfaces (depth SELECT, depth CALIBRATE, depth LEARN) in a within-subject user study (n=14), showing that steering can align outputs with latent user preferences and that interface design influences perceived control, usability, and transparency. The results highlight steering as a lightweight, privacy-conscious, and flexible personalization framework with practical potential for real-world, multi-turn discussions, including AR/VR contexts, while acknowledging calibration and model-sensitivity limitations. Overall, the work demonstrates that inference-time preference steering can significantly improve alignment with user values in a scalable, low-overhead manner.

Abstract

As large language models (LLMs) improve in their capacity to serve as personal AI assistants, their ability to output uniquely tailored, personalized responses that align with the soft preferences of their users is essential for enhancing user satisfaction and retention. However, untrained lay users have poor prompt specification abilities and often struggle with conveying their latent preferences to AI assistants. To address this, we leverage activation steering to guide LLMs to align with interpretable preference dimensions during inference. In contrast to memory-based personalization methods that require longer user history, steering is extremely lightweight and can be easily controlled by the user via an linear strength factor. We embed steering into three different interactive chatbot interfaces and conduct a within-subjects user study (n=14) to investigate how end users prefer to personalize their conversations. The results demonstrate the effectiveness of preference-based steering for aligning real-world conversations with hidden user preferences, and highlight further insights on how diverse values around control, usability, and transparency lead users to prefer different interfaces.

Figures (18)

• Figure 1: Effect of expressed preferences (top) and perplexity-normalized effect (bottom) for all preferences and models for E1.
• Figure 2: Effect of additional prompting on steering for two preferences: cost (top) and age (bottom) for E2.
• Figure 3: Preference effects for culture and age when both are steered additively for Mistral-7B-Instruct-v0.3 in E3.
• Figure 4: Results for gemma-2-2b-it and Qwen2.5-7B-Instruct for learning preferences in E5, with estimated preference steering strength (top) and corresponding cost effect sizes (bottom). Blue represents trials where the latent preference $h$ is 100% budget, and red represents 100% luxury. .
• Figure 5: Overview of the within-subjects user study procedure, where participants completed a different personalization task using each of the four interfaces. Participants recorded their true preferences before the tasks and their perceptions afterwards.