Understanding the Role of User Profile in the Personalization of Large Language Models

TL;DR

This study first confirms that the effectiveness of user profiles is primarily due to personalization information rather than semantic information, and investigates how user profiles affect the personalization of LLMs.

Abstract

Utilizing user profiles to personalize Large Language Models (LLMs) has been shown to enhance the performance on a wide range of tasks. However, the precise role of user profiles and their effect mechanism on LLMs remains unclear. This study first confirms that the effectiveness of user profiles is primarily due to personalization information rather than semantic information. Furthermore, we investigate how user profiles affect the personalization of LLMs. Within the user profile, we reveal that it is the historical personalized response produced or approved by users that plays a pivotal role in personalizing LLMs. This discovery unlocks the potential of LLMs to incorporate a greater number of user profiles within the constraints of limited input length. As for the position of user profiles, we observe that user profiles integrated into different positions of the input context do not contribute equally to personalization. Instead, where the user profile that is closer to the beginning affects more on the personalization of LLMs. Our findings reveal the role of user profiles for the personalization of LLMs, and showcase how incorporating user profiles impacts performance providing insight to leverage user profiles effectively.

Figures (9)

• Figure 1: Our key findings: (a) Semantic information is less critical to the effectiveness of user profile than personalization, and it only contributes when built on top of personalization (§\ref{['sec:actual_affect']}); (b) The impact of user responses on personalization is greater than that of previous input and their mapping between users' previous input and response (§\ref{['subsec: single user profiles']}); (c) The user profiles in the different positions of the context contribute differently to the personalization, where profiles positioned closer to the beginning contribute more (§\ref{['subsec: the order']}).
• Figure 2: The improvement of performance (Flan-T5-base) on LaMP dataset with different Augmentation based on the user profiles ($k=1$) compared to without augmentations. Note that LaMP-3 shows a decreases in performance compared to the no-augmentations baseline, indicated by the lower values of both MAE and RMSE, where lower scores signify better performance.
• Figure 3: The performance on 4 LaMP tasks, with different combinations of sampling strategies for the construction of the introduced user profiles. Note that the metrics for LaMP-3 are the lower, the better.
• Figure 4: The performance with different numbers of used user profiles on four LaMP datasets when only with the input part and only with the output part. Note that the metrics for LaMP-3 are the lower, the better.
• Figure 5: The performance with different proportions of user profiles on four LaMP datasets when only using the output part of the completed user profile. Note that the metrics for LaMP-3 are the lower, the better.