Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Personalized Adaptation via In-Context Preference Learning

Allison Lau, Younwoo Choi, Vahid Balazadeh, Keertana Chidambaram, Vasilis Syrgkanis, Rahul G. Krishnan

TL;DR

The Preference Pretrained Transformer is presented, a novel approach for adaptive personalization using online user feedback that leverages the in-context learning capabilities of transformers to dynamically adapt to individual preferences.

Abstract

Reinforcement Learning from Human Feedback (RLHF) is widely used to align Language Models (LMs) with human preferences. However, existing approaches often neglect individual user preferences, leading to suboptimal personalization. We present the Preference Pretrained Transformer (PPT), a novel approach for adaptive personalization using online user feedback. PPT leverages the in-context learning capabilities of transformers to dynamically adapt to individual preferences. Our approach consists of two phases: (1) an offline phase where we train a single policy model using a history-dependent loss function, and (2) an online phase where the model adapts to user preferences through in-context learning. We demonstrate PPT's effectiveness in a contextual bandit setting, showing that it achieves personalized adaptation superior to existing methods while significantly reducing the computational costs. Our results suggest the potential of in-context learning for scalable and efficient personalization in large language models.

Personalized Adaptation via In-Context Preference Learning

TL;DR

The Preference Pretrained Transformer is presented, a novel approach for adaptive personalization using online user feedback that leverages the in-context learning capabilities of transformers to dynamically adapt to individual preferences.

Abstract

Reinforcement Learning from Human Feedback (RLHF) is widely used to align Language Models (LMs) with human preferences. However, existing approaches often neglect individual user preferences, leading to suboptimal personalization. We present the Preference Pretrained Transformer (PPT), a novel approach for adaptive personalization using online user feedback. PPT leverages the in-context learning capabilities of transformers to dynamically adapt to individual preferences. Our approach consists of two phases: (1) an offline phase where we train a single policy model using a history-dependent loss function, and (2) an online phase where the model adapts to user preferences through in-context learning. We demonstrate PPT's effectiveness in a contextual bandit setting, showing that it achieves personalized adaptation superior to existing methods while significantly reducing the computational costs. Our results suggest the potential of in-context learning for scalable and efficient personalization in large language models.

Paper Structure

This paper contains 6 sections, 8 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Preference Pretrained Transformer
  3. Experiments
  4. Experimental Setup
  5. Results
  6. Conclusion

Figures (2)

  • Figure 1: Preference Pretrained Transformer: (i) In the offline phase, we train a single policy to predict the preferred answers given the history of previous responses. (ii) In the online phase, the pretrained model interacts with the user, appends the interaction history to its context and generates more personalized responses.
  • Figure 2: Comparison of rewards between PPT (ours) and the Personalized Soups (PS) over 15 interaction turns for different user groups. Figure \ref{['fig:rewards_500']} and Figure \ref{['fig:rewards_1000']} show results with $N_c = 500$ and $N_c = 1000$ context vectors, respectively. Each subplot corresponds to tests with users from one of the three subpopulations and a user with mixed preferences. The results demonstrate that PPT consistently outperforms the PS baseline across all groups. The increase in rewards for our method as the number of turns grows indicates effective in-context learning and dynamic adaptation to user preferences.