"Are You Sure?": An Empirical Study of Human Perception Vulnerability in LLM-Driven Agentic Systems

TL;DR

This work presents the first large-scale empirical study with 303 participants to measure human susceptibility to Agent-Mediated Deception, and identifies six cognitive failure modes in users and finds that their risk awareness often fails to translate to protective behavior.

Abstract

Large language model (LLM) agents are rapidly becoming trusted copilots in high-stakes domains like software development and healthcare. However, this deepening trust introduces a novel attack surface: Agent-Mediated Deception (AMD), where compromised agents are weaponized against their human users. While extensive research focuses on agent-centric threats, human susceptibility to deception by a compromised agent remains unexplored. We present the first large-scale empirical study with 303 participants to measure human susceptibility to AMD. This is based on HAT-Lab (Human-Agent Trust Laboratory), a high-fidelity research platform we develop, featuring nine carefully crafted scenarios spanning everyday and professional domains (e.g., healthcare, software development, human resources). Our 10 key findings reveal significant vulnerabilities and provide future defense perspectives. Specifically, only 8.6% of participants perceive AMD attacks, while domain experts show increased susceptibility in certain scenarios. We identify six cognitive failure modes in users and find that their risk awareness often fails to translate to protective behavior. The defense analysis reveals that effective warnings should interrupt workflows with low verification costs. With experiential learning based on HAT-Lab, over 90% of users who perceive risks report increased caution against AMD. This work provides empirical evidence and a platform for human-centric agent security research.

Figures (11)

• Figure 1: Examples of stealthy LLM agent-mediated deception. (1) An agent manipulated by a hidden note in a resume biases the hiring decision. (2) An agent hijacked by a malicious email exfiltrates sensitive data and deletes the evidence.
• Figure 2: System overview of HAT-Lab.
• Figure 3: Illustrations of three guardrail conditions.
• Figure 4: The three-phase procedure of our experiment. Participants are randomly assigned to a guardrail in Phase 2.
• Figure 5: Pre-survey attitudes towards LLM agents (N=303). Users have a mixed, contradictory view. (a) 97.0% have a moderate to high trust in AI to do tasks correctly. (b) 75.9% say they check AI outputs often. (c) 96.4% have a moderate to very high confidence in identifying risks in AI systems. (d) Only 28.4% report they never experience AI issues.