Beyond Fixed Psychological Personas: State Beats Trait, but Language Models are State-Blind

TL;DR

Chameleon introduces a dataset of 5,001 contextual psychological profiles from 1,667 Reddit users across 645 subreddits to operationalize Latent State-Trait theory in NLP. The study demonstrates that $ICC \approx 0.26$–$0.28$, implying $72\%-74\%$ of variance in expressed psychology is within-person (state) rather than between-person (trait), and validates this with two extraction methods yielding high profile-level agreement ($r \approx 0.71$). It further shows that LLMs are state-blind in generation, while reward models are context-aware but inconsistent in direction, with a notable “vulnerable user paradox” impacting RLHF. The Chameleon dataset and its accompanying pipeline enable principled investigation of psychology-aware AI, revealing critical implications for personalization, evaluation fairness, and RLHF alignment. The work motivates future research to develop state-sensitive generation and stable, state-aware evaluation mechanisms across diverse platforms and modalities.

Abstract

User interactions with language models vary due to static properties of the user (trait) and the specific context of the interaction (state). However, existing persona datasets (like PersonaChat, PANDORA etc.) capture only trait, and ignore the impact of state. We introduce Chameleon, a dataset of 5,001 contextual psychological profiles from 1,667 Reddit users, each measured across multiple contexts. Using the Chameleon dataset, we present three key findings. First, inspired by Latent State-Trait theory, we decompose variance and find that 74\% is within-person(state) while only 26\% is between-person (trait). Second, we find that LLMs are state-blind: they focus on trait only, and produce similar responses regardless of state. Third, we find that reward models react to user state, but inconsistently: different models favor or penalize the same users in opposite directions. We release Chameleon to support research on affective computing, personalized dialogue, and RLHF alignment.

Figures (9)

• Figure 1: AI systems get psychological context backwards. The same user (John) expresses different psychological states across contexts (74% of variance is within-person). (A) Generation: LLMs produce nearly identical responses regardless of user profile. (B) Evaluation: Reward models score identical responses differently based on user profile, but disagree on direction. LLMs are state-blind; reward models are context-aware but inconsistent.
• Figure 2: Chameleon profile extraction pipeline. Each post is processed through two parallel extraction methods (SEANCEcrossley2017sentiment and LangExtractLangExtract2025), assessed against 26 psychological scales via LLM, z-normalized, and fused into a final profile.
• Figure 3: Cross-method profile agreement. Left: Distribution of within-post correlations between SEANCE and LangExtract profiles (mean $r$ = .71, median = .76). Right: Cumulative distribution showing 69.9% of posts achieve $r > .70$ (high agreement) and 91.8% achieve $r > .50$.
• Figure 4: MTMM correlation matrix. Rows: SEANCE scales. Columns: LangExtract scales. Yellow borders indicate diagonal (convergent validity).
• Figure 5: Distribution of psychological dimension scores. SEANCE (blue), LangExtract (green), and fused after z-normalization (red).