Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DarkPatterns-LLM: A Multi-Layer Benchmark for Detecting Manipulative and Harmful AI Behavior

Sadia Asif, Israel Antonio Rosales Laguan, Haris Khan, Shumaila Asif, Muneeb Asif

TL;DR

DarkPatterns-LLM tackles the problem of subtle, manipulative AI behavior by introducing a multi-layer, explainable benchmark that transcends binary safety labels. The four-layer pipeline—MGD, MSAIN, THP, and DCRA—provides fine-grained diagnostics of manipulation across seven harm categories, tracking not only immediate risk but also stakeholder impact and temporal dynamics. The dataset of 401 curated instruction–response examples, along with expert annotations, enables systematic evaluation of six state-of-the-art LLMs, revealing significant performance gaps, particularly in autonomy-harm detection and long-horizon risk modeling. This benchmark advances trustworthy AI by enabling interpretable safety assessments, guiding targeted mitigations, and laying groundwork for scalable, multi-domain safety monitoring in real-world deployments.

Abstract

The proliferation of Large Language Models (LLMs) has intensified concerns about manipulative or deceptive behaviors that can undermine user autonomy, trust, and well-being. Existing safety benchmarks predominantly rely on coarse binary labels and fail to capture the nuanced psychological and social mechanisms constituting manipulation. We introduce \textbf{DarkPatterns-LLM}, a comprehensive benchmark dataset and diagnostic framework for fine-grained assessment of manipulative content in LLM outputs across seven harm categories: Legal/Power, Psychological, Emotional, Physical, Autonomy, Economic, and Societal Harm. Our framework implements a four-layer analytical pipeline comprising Multi-Granular Detection (MGD), Multi-Scale Intent Analysis (MSIAN), Threat Harmonization Protocol (THP), and Deep Contextual Risk Alignment (DCRA). The dataset contains 401 meticulously curated examples with instruction-response pairs and expert annotations. Through evaluation of state-of-the-art models including GPT-4, Claude 3.5, and LLaMA-3-70B, we observe significant performance disparities (65.2\%--89.7\%) and consistent weaknesses in detecting autonomy-undermining patterns. DarkPatterns-LLM establishes the first standardized, multi-dimensional benchmark for manipulation detection in LLMs, offering actionable diagnostics toward more trustworthy AI systems.

DarkPatterns-LLM: A Multi-Layer Benchmark for Detecting Manipulative and Harmful AI Behavior

TL;DR

DarkPatterns-LLM tackles the problem of subtle, manipulative AI behavior by introducing a multi-layer, explainable benchmark that transcends binary safety labels. The four-layer pipeline—MGD, MSAIN, THP, and DCRA—provides fine-grained diagnostics of manipulation across seven harm categories, tracking not only immediate risk but also stakeholder impact and temporal dynamics. The dataset of 401 curated instruction–response examples, along with expert annotations, enables systematic evaluation of six state-of-the-art LLMs, revealing significant performance gaps, particularly in autonomy-harm detection and long-horizon risk modeling. This benchmark advances trustworthy AI by enabling interpretable safety assessments, guiding targeted mitigations, and laying groundwork for scalable, multi-domain safety monitoring in real-world deployments.

Abstract

The proliferation of Large Language Models (LLMs) has intensified concerns about manipulative or deceptive behaviors that can undermine user autonomy, trust, and well-being. Existing safety benchmarks predominantly rely on coarse binary labels and fail to capture the nuanced psychological and social mechanisms constituting manipulation. We introduce \textbf{DarkPatterns-LLM}, a comprehensive benchmark dataset and diagnostic framework for fine-grained assessment of manipulative content in LLM outputs across seven harm categories: Legal/Power, Psychological, Emotional, Physical, Autonomy, Economic, and Societal Harm. Our framework implements a four-layer analytical pipeline comprising Multi-Granular Detection (MGD), Multi-Scale Intent Analysis (MSIAN), Threat Harmonization Protocol (THP), and Deep Contextual Risk Alignment (DCRA). The dataset contains 401 meticulously curated examples with instruction-response pairs and expert annotations. Through evaluation of state-of-the-art models including GPT-4, Claude 3.5, and LLaMA-3-70B, we observe significant performance disparities (65.2\%--89.7\%) and consistent weaknesses in detecting autonomy-undermining patterns. DarkPatterns-LLM establishes the first standardized, multi-dimensional benchmark for manipulation detection in LLMs, offering actionable diagnostics toward more trustworthy AI systems.
Paper Structure (29 sections, 10 equations, 1 figure, 3 tables)

This paper contains 29 sections, 10 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Layer 1: Multi-Granular Detection (MGD)
  5. Layer 2: Multi-Scale Intent Analysis (MSIAN)
  6. Layer 2: Multi-Scale Intent Analysis (MSIAN)
  7. Layer 3: Threat Harmonization Protocol (THP)
  8. Layer 4: Deep Contextual Risk Alignment (DCRA)
  9. Dataset Construction
  10. Harm Taxonomy
  11. Data Collection Pipeline
  12. Dataset Statistics
  13. Experiments
  14. Experimental Setup
  15. Results
  16. ...and 14 more sections

Figures (1)

  • Figure 1: Dataset distribution across harm categories in the DarkPatterns-LLM corpus. The dataset contains 401 total entries across seven harm categories, with proportions ranging from 12.0% to 17.2%.