Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Personalize Your LLM: Fake it then Align it

Yijing Zhang, Dyah Adila, Changho Shin, Frederic Sala

TL;DR

CHAMELEON tackles scalable LLM personalization by coupling self-generated synthetic user preference data with representation editing, enabling per-user alignment without fine-tuning or large labeled datasets. It learns personalized and non-personalized embedding subspaces (via SVD and CCS) and applies inference-time edits to enhance user-specific behavior, while also enabling efficient group-scale personalization. Empirical results on the LaMP benchmark show CHAMELEON outperforms two personalization baselines and a time-constrained DPO setup, with average gains around $40\%$ across architectures, and demonstrated generalization to unseen users. The approach offers data- and compute-efficient personalization with practical implications for deploying personalized LLMs at scale, albeit with limitations tied to the quality of self-generated data and privacy considerations.

Abstract

Personalizing large language models (LLMs) is essential for delivering tailored interactions that improve user experience. Many existing personalization methods require fine-tuning LLMs for each user, rendering them prohibitively expensive for widespread adoption. Although retrieval-based approaches offer a more compute-efficient alternative, they still depend on large, high-quality datasets that are not consistently available for all users. To address this challenge, we propose CHAMELEON, a scalable and efficient personalization approach that uses (1) self-generated personal preference data and (2) representation editing to enable quick and cost-effective personalization. Our experiments on various tasks, including those from the LaMP personalization benchmark, show that CHAMELEON efficiently adapts models to personal preferences, improving instruction-tuned models and outperforms two personalization baselines by an average of 40% across two model architectures.

Personalize Your LLM: Fake it then Align it

TL;DR

CHAMELEON tackles scalable LLM personalization by coupling self-generated synthetic user preference data with representation editing, enabling per-user alignment without fine-tuning or large labeled datasets. It learns personalized and non-personalized embedding subspaces (via SVD and CCS) and applies inference-time edits to enhance user-specific behavior, while also enabling efficient group-scale personalization. Empirical results on the LaMP benchmark show CHAMELEON outperforms two personalization baselines and a time-constrained DPO setup, with average gains around across architectures, and demonstrated generalization to unseen users. The approach offers data- and compute-efficient personalization with practical implications for deploying personalized LLMs at scale, albeit with limitations tied to the quality of self-generated data and privacy considerations.

Abstract

Personalizing large language models (LLMs) is essential for delivering tailored interactions that improve user experience. Many existing personalization methods require fine-tuning LLMs for each user, rendering them prohibitively expensive for widespread adoption. Although retrieval-based approaches offer a more compute-efficient alternative, they still depend on large, high-quality datasets that are not consistently available for all users. To address this challenge, we propose CHAMELEON, a scalable and efficient personalization approach that uses (1) self-generated personal preference data and (2) representation editing to enable quick and cost-effective personalization. Our experiments on various tasks, including those from the LaMP personalization benchmark, show that CHAMELEON efficiently adapts models to personal preferences, improving instruction-tuned models and outperforms two personalization baselines by an average of 40% across two model architectures.

Paper Structure

This paper contains 56 sections, 7 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Personalized LLMs.
  4. Representation Editing for Personalization.
  5. Chameleon: Personalization through Representation Editing
  6. Self-generated Preference Data
  7. User History Selection.
  8. Insight Generation.
  9. Generating Synthetic User Preference Data
  10. Representation Editing
  11. Group-scale Personalization
  12. Experiments
  13. Experimental Setup
  14. Datasets and Tasks.
  15. Evaluation Metrics.
  16. ...and 41 more sections

Figures (5)

  • Figure 1: $\textsc{Chameleon}$ identifies two separate subspaces, one personalized and one non-personalized, from self-generated user characteristic insights. Based on these subspaces, we modify the LLM embeddings during inference.
  • Figure 2: Self-generated user preference data: we use the generated conclusion of user characteristics to guide the personal answer generation.
  • Figure 3: The change of performance when different number of users are given to $\textsc{Chameleon}$
  • Figure 4: $\textsc{Chameleon}$ compared with DPO and ALOE in time-constrained scenarios. The columns denotes the improvement from the instruction-tuned model.
  • Figure 5: The change of performance when different number of history data per user are given to $\textsc{Chameleon}$