Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can LLMs Discern the Traits Influencing Your Preferences? Evaluating Personality-Driven Preference Alignment in LLMs

Tianyu Zhao, Siqi Li, Yasser Shoukry, Salma Elmalaki

TL;DR

This work addresses the brittleness of preference-based personalization in LLMs by introducing PACIFIC, a psychometrically grounded dataset that links stated user preferences to Big Five personality traits. The authors demonstrate that aligning preferences with inferred personality signals substantially improves answer accuracy, achieving large gains over baselines. They present a four-method framework, including explicit trait labeling and retrieval-augmented prompting, and show that explicit cues yield the strongest performance while RAG offers a practical alternative when trait annotations are unavailable. The work highlights the potential of using personality as a compact, robust memory abstraction for user modeling and discusses implications for controllable, user-aligned generation, along with observed biases and future research directions.

Abstract

User preferences are increasingly used to personalize Large Language Model (LLM) responses, yet how to reliably leverage preference signals for answer generation remains under-explored. In practice, preferences can be noisy, incomplete, or even misleading, which can degrade answer quality when applied naively. Motivated by the observation that stable personality traits shape everyday preferences, we study personality as a principled ''latent'' signal behind preference statements. Through extensive experiments, we find that conditioning on personality-aligned preferences substantially improves personalized question answering: selecting preferences consistent with a user's inferred personality increases answer-choice accuracy from 29.25% to 76%, compared to using randomly selected preferences. Based on these findings, we introduce PACIFIC (Preference Alignment Choices Inference for Five-factor Identity Characterization), a personality-labeled preference dataset containing 1200 preference statements spanning diverse domains (e.g., travel, movies, education), annotated with Big-Five (OCEAN) trait directions. Finally, we propose a framework that enables an LLM model to automatically retrieve personality-aligned preferences and incorporate them during answer generation.

Can LLMs Discern the Traits Influencing Your Preferences? Evaluating Personality-Driven Preference Alignment in LLMs

TL;DR

This work addresses the brittleness of preference-based personalization in LLMs by introducing PACIFIC, a psychometrically grounded dataset that links stated user preferences to Big Five personality traits. The authors demonstrate that aligning preferences with inferred personality signals substantially improves answer accuracy, achieving large gains over baselines. They present a four-method framework, including explicit trait labeling and retrieval-augmented prompting, and show that explicit cues yield the strongest performance while RAG offers a practical alternative when trait annotations are unavailable. The work highlights the potential of using personality as a compact, robust memory abstraction for user modeling and discusses implications for controllable, user-aligned generation, along with observed biases and future research directions.

Abstract

User preferences are increasingly used to personalize Large Language Model (LLM) responses, yet how to reliably leverage preference signals for answer generation remains under-explored. In practice, preferences can be noisy, incomplete, or even misleading, which can degrade answer quality when applied naively. Motivated by the observation that stable personality traits shape everyday preferences, we study personality as a principled ''latent'' signal behind preference statements. Through extensive experiments, we find that conditioning on personality-aligned preferences substantially improves personalized question answering: selecting preferences consistent with a user's inferred personality increases answer-choice accuracy from 29.25% to 76%, compared to using randomly selected preferences. Based on these findings, we introduce PACIFIC (Preference Alignment Choices Inference for Five-factor Identity Characterization), a personality-labeled preference dataset containing 1200 preference statements spanning diverse domains (e.g., travel, movies, education), annotated with Big-Five (OCEAN) trait directions. Finally, we propose a framework that enables an LLM model to automatically retrieve personality-aligned preferences and incorporate them during answer generation.
Paper Structure (42 sections, 4 equations, 4 figures, 10 tables)

This paper contains 42 sections, 4 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. PACIFIC Dataset
  3. Problem Definition
  4. Dual-Strategy Annotation
  5. The Mirroring Strategy (O, C, E, A)
  6. The Compensatory Strategy (N)
  7. Confidence Creteria
  8. Dataset Statistics and Quality Control
  9. Persona-Driven Preference-Following Framework
  10. Motivation & Preliminaries
  11. Personality Alignment Methods
  12. Few-shot (Preference-only).
  13. Few Shot + Persona Trait Hints
  14. Reminder
  15. Retrieval-Augmented Generation (RAG)
  16. ...and 27 more sections

Figures (4)

  • Figure 1: Overview of the PACIFIC dataset. The dataset encompasses diverse user interactions across 20 distinct topics, including Home Cooking, Personal Finance, and Moving Relocation. It features 10 persona traits derived from the OCEAN model: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. The figure illustrates a sample conversation where an agent provides options tailored to a user's high-openness preference. Each turn is accompanied by a Persona trait label (together with confidence score) that reflects the persona alignment of both the user's expressed preference and each of the agent's provided choices.
  • Figure 2: Method overview for persona-aware preference prompting. Given a user conversation and a multiple-choice question, we first infer the user’s OCEAN persona profile from their preference statements (left). We then construct the model input in four different ways (right): (i) Few-shot, which includes a fixed set of preference examples; (ii) Persona trait + few-shot, which augments the same examples with an explicit persona profile; (iii) Reminder, which adds a brief instruction to consider the inferred persona when answering; and (iv) Retrieval-Augmented Generation (RAG), which retrieves persona-consistent preferences by embedding the question and candidate preferences and ranking them by similarity, then inserts the top retrieved preferences into the prompt. The model outputs a single selected choice (bottom).
  • Figure 3: Traits distribution overview: The preference in PrefEval demonstrated distribution skew where some traits(Low-C, High-E, Low-A and Low-N) preferences are limited. In total, PrefEval has 613 personality-driven preferences out of 1000 preferences with confidence of $\tau = 0.7$. PACIFIC yields 803, 746, 574 personality-driven preferences out of 1200 preferences with confidence of $\tau = 0.7, 0.8, 0.9$ respectively.
  • Figure 4: Accuracy (%) by persona group (32 total). Each panel corresponds to one group (Idx 0–31) and shows the overall accuracy plus per-trait accuracies for the Big Five (O, C, E, A, N). Bar colors encode traits (Overall in magenta with white hatch; O/C/E/A/N in distinct colors), and titles indicate each group’s high/low trait configuration