Who's in Charge? Disempowerment Patterns in Real-World LLM Usage

TL;DR

This paper provides the first large-scale empirical analysis of disempowerment patterns in real-world AI-assisted usage, focusing on situational disempowerment defined through distortions of reality, value judgments, or actions. It introduces a framework with three disempowerment primitives and four amplifying factors, and uses privacy-preserving methods to measure potential and actualization across 1.5 million Claude.ai conversations. The study finds severe disempowerment is rare but domains involving personal decisions show higher risk, and amplifying factors monotonically relate to increased disempowerment potential and actualization; it also reveals that users often rate disempowering interactions positively, raising concerns about training signals. The work further shows mixed evidence that current preference models avoid disempowerment, emphasizing the need for empowerment-focused design and benchmarks to promote human flourishing in AI-enabled society.

Abstract

Although AI assistants are now deeply embedded in society, there has been limited empirical study of how their usage affects human empowerment. We present the first large-scale empirical analysis of disempowerment patterns in real-world AI assistant interactions, analyzing 1.5 million consumer Claude$.$ai conversations using a privacy-preserving approach. We focus on situational disempowerment potential, which occurs when AI assistant interactions risk leading users to form distorted perceptions of reality, make inauthentic value judgments, or act in ways misaligned with their values. Quantitatively, we find that severe forms of disempowerment potential occur in fewer than one in a thousand conversations, though rates are substantially higher in personal domains like relationships and lifestyle. Qualitatively, we uncover several concerning patterns, such as validation of persecution narratives and grandiose identities with emphatic sycophantic language, definitive moral judgments about third parties, and complete scripting of value-laden personal communications that users appear to implement verbatim. Analysis of historical trends reveals an increase in the prevalence of disempowerment potential over time. We also find that interactions with greater disempowerment potential receive higher user approval ratings, possibly suggesting a tension between short-term user preferences and long-term human empowerment. Our findings highlight the need for AI systems designed to robustly support human autonomy and flourishing.

Figures (28)

• Figure 1: Representative empirical results.(Left) A privacy-preserving cluster summary describing a pattern of action distortion potential, in which users repeatedly delegate romantic communications to the AI and appear to implement responses verbatim. (Middle) A privacy-preserving cluster summary describing a pattern of authority projection, in which users position the AI as a dominant authority figure across sustained interactions. The cluster summaries are produced by our analysis pipeline and lightly edited for brevity. (Right) The prevalence of moderate-or-severe disempowerment potential and amplifying factors in user-submitted feedback data over time, showing an apparent increase throughout the observation period. Multiple factors may explain this trend, including potential shifts in the composition of users providing feedback and changes in user trust in AI over time. No causal attribution to any specific model version is supported by this observational data.
• Figure 2: Prevalence of disempowerment potential primitives and amplifying factors among randomly sampled Claude.ai interactions. Reality distortion potential is the most common severe-level primitive, while vulnerability is the most prevalent severe-level amplifying factor. All primitives and amplifying factors classified as severe occur at rates exceeding 1 in 10,000 interactions. Error bars indicate 95% confidence intervals calculated using the Wilson score method.
• Figure 3: Domain-specific analysis of disempowerment patterns.(a) Disempowerment potential rate (any primitive rate moderate or severe) varies substantially across domains. Relationships & Lifestyle shows the highest rate ($\sim$8%), followed by Society & Culture and Healthcare & Wellness ($\sim$5% each), while technical domains like Software Development show rates below 1%. (b) Actualized disempowerment rates follow a broadly similar pattern, though at much lower absolute values. (c) Domain prevalence reveals an inverse relationship with disempowerment risk: Technical domains like Software Development dominate overall usage ($\sim$40% of interactions) but pose minimal disempowerment risk, while high-risk domains collectively represent a smaller share of traffic. Error bars indicate 95% confidence intervals calculated using the Wilson score method.
• Figure 4: Amplifying factors and disempowerment actualization.(Top row) Rate of any moderate or severe disempowerment potential primitive across amplifying factor severity levels. All the amplifying factors are associated with increased disempowerment potential, with both 'reliance and dependency' and 'vulnerabilty' showing clear monotonic relationships. (Bottom row) Disempowerment actualization rates i.e., the proportion of interactions with at least moderate disempowerment potential that also show signs of actualized disempowerment—conditioned on amplifying factor severity. We find consistent monotonic relationships across all amplifying factors i.e., the amplifying factors are associated with increased disempowerment actualization, with authority projection and attachment having the largest associations. Error bars indicate 95% confidence intervals using the Wilson score method.
• Figure 5: Understanding reality distortion potential.(Top) Illustrative cluster summaries of severe reality distortion potential: validation of persecution narratives involving surveillance and coordinated targeting (left), and validation of grandiose spiritual identity claims (right). Both clusters exhibit similar dynamics—emphatic AI validation, unfalsifiable user frameworks, and escalating elaboration over multiple exchanges. (Bottom) Quantitative analysis of 132 cluster descriptions derived from 7,200 conversations with moderate or severe reality distortion potential. We estimate the frequency of various factors within conversations among all conversations with moderate or severe reality distortion potential. (a) Distortion mechanisms. Sycophantic validation is the most prevalent, while outright fabrication is rare. (b) Distortion targets: third-party mental states are most common, but all occur frequently. (c) User behavior: most users actively build upon AI-validated beliefs and seek validation, while reality testing is rare. (d) Conversational trajectory: the majority of conversations exhibit escalating distortion potential over the course of the conversation. Error bars indicate 95% confidence intervals calculated using the Wilson score method.