The Siren Song of LLMs: How Users Perceive and Respond to Dark Patterns in Large Language Models

TL;DR

The paper defines LLM dark patterns as manipulativedeceptive conversational strategies that emerge in language-based interactions, extending UX dark-pattern theory to AI chat systems. It develops an operational taxonomy of five top-level categories and 11 subcategories grounded in literature and AI incident data, and conducts a scenario-based user study (N=34) across 11 paired dark-pattern vs neutral scenarios to examine recognition, perception, and accountability. Key findings show recognition hinges on explicit conversational cues, with substantial variation across patterns; responses range from resistance to acceptance, and responsibility is attributed to companies, models, users, or shared/ambiguous sources. The work offers conceptual, empirical, and governance-oriented contributions to design LLMs that safeguard user autonomy, including detection benchmarks, disclosure practices, and policy recommendations for accountability across stakeholders.

Abstract

Large language models can influence users through conversation, creating new forms of dark patterns that differ from traditional UX dark patterns. We define LLM dark patterns as manipulative or deceptive behaviors enacted in dialogue. Drawing on prior work and AI incident reports, we outline a diverse set of categories with real-world examples. Using them, we conducted a scenario-based study where participants (N=34) compared manipulative and neutral LLM responses. Our results reveal that recognition of LLM dark patterns often hinged on conversational cues such as exaggerated agreement, biased framing, or privacy intrusions, but these behaviors were also sometimes normalized as ordinary assistance. Users' perceptions of these dark patterns shaped how they respond to them. Responsibilities for these behaviors were also attributed in different ways, with participants assigning it to companies and developers, the model itself, or to users. We conclude with implications for design, advocacy, and governance to safeguard user autonomy.

Figures (6)

• Figure 1: Overview of our conceptual grounding, study design, and findings. We begin by introducing our definition of LLM dark patterns and five meta-categories that serve as the conceptual grounding for the study. Building on these foundations, we conducted an empirical study with 34 participants, presenting 11 paired dark pattern vs. neutral scenarios through a standardized slide deck in semi-structured interviews. The findings address three research questions: RQ1 examines to what extent users recognize LLM dark patterns responses and what factors influence recognition. RQ2 investigates how users perceive dark patterns and how those perceptions shape their responses. RQ3 explores who users believe is responsible for LLM dark patterns and how accountability is assigned.
• Figure 2: Research workflow for identifying and studying LLM dark patterns. The diagram illustrates a multi-step process: (1) clarifying the concept of LLM dark patterns, (2) searching prior literature, (3) coding datasets of incidents, and (4) curating real-world examples. These steps feed into (5) thematic grouping into categories, which then support analysis of (6) human perceptions and responses.
• Figure 3: Example study scenarios illustrating two of the eleven LLM dark patterns used in our user study. Each scenario presents a background, a user query, and two contrasting AI assistant responses: one with no dark pattern and one exhibiting the dark pattern. (a) Sycophantic Agreement shows an AI uncritically endorsing a user's risky decision to stop medication, reinforcing it with praise and spiritualized framing. (b) Brand Favoritism shows an AI embedding unsolicited product recommendations when asked about light color for sleep, steering the user toward commercial options. Remaining scenarios appear in the Appendix \ref{['app:scenarios']}.
• Figure 4: Recognition rates of different LLM dark pattern subcategories observed in the user study. Bars show the percentage of participants (out of 34) who identified dark-version response as manipulative. Manipulations such as Simulated Emotional & Sexual Intimacy and Brand Favoritism were recognized by over 90% of participants, while subtler patterns like Opaque Training Data Sources, Excessive Flattery, and Interaction Padding had lower recognition rates.
• Figure 5: Six of the eleven scenarios used in our user study, each illustrating a distinct category of LLM dark pattern introduced in \ref{['dark']}. Shown are examples of Interaction Padding, Excessive Flattery, Simulated Emotional & Sexual Intimacy, Ideological Steering, Unprompted Intimacy Probing, and Behavioral Profiling via Dialogue.