Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WOLF: Werewolf-based Observations for LLM Deception and Falsehoods

Mrinal Agarwal, Saad Rana, Theo Sundoro, Hermela Berhe, Spencer Kim, Vasu Sharma, Sean O'Brien, Kevin Zhu

TL;DR

WOLF introduces a dynamic, statement-level deception benchmark based on Werewolf to jointly measure deception production by agents and deception detection by peers in a controllable multi-agent setting. By embedding role-grounded agents in a LangGraph with strict night-day cycles, WOLF captures longitudinal trust dynamics through self- and peer-assessments, logging every event for reproducibility. Across 100 games and more than 7,200 analyzed statements, the results reveal a robust deception production by Werewolves (about 31% of turns) and a moderate but imperfect detection performance (precision ~71–73%, recall ~48–61%), with suspicion converging more strongly on deceivers over time. The framework provides a principled, auditable testbed that moves deception research beyond static datasets toward interactive, adversarial dynamics with actionable insights for calibration and detection strategies in LLMs.

Abstract

Deception is a fundamental challenge for multi-agent reasoning: effective systems must strategically conceal information while detecting misleading behavior in others. Yet most evaluations reduce deception to static classification, ignoring the interactive, adversarial, and longitudinal nature of real deceptive dynamics. Large language models (LLMs) can deceive convincingly but remain weak at detecting deception in peers. We present WOLF, a multi-agent social deduction benchmark based on Werewolf that enables separable measurement of deception production and detection. WOLF embeds role-grounded agents (Villager, Werewolf, Seer, Doctor) in a programmable LangGraph state machine with strict night-day cycles, debate turns, and majority voting. Every statement is a distinct analysis unit, with self-assessed honesty from speakers and peer-rated deceptiveness from others. Deception is categorized via a standardized taxonomy (omission, distortion, fabrication, misdirection), while suspicion scores are longitudinally smoothed to capture both immediate judgments and evolving trust dynamics. Structured logs preserve prompts, outputs, and state transitions for full reproducibility. Across 7,320 statements and 100 runs, Werewolves produce deceptive statements in 31% of turns, while peer detection achieves 71-73% precision with ~52% overall accuracy. Precision is higher for identifying Werewolves, though false positives occur against Villagers. Suspicion toward Werewolves rises from ~52% to over 60% across rounds, while suspicion toward Villagers and the Doctor stabilizes near 44-46%. This divergence shows that extended interaction improves recall against liars without compounding errors against truthful roles. WOLF moves deception evaluation beyond static datasets, offering a dynamic, controlled testbed for measuring deceptive and detective capacity in adversarial multi-agent interaction.

WOLF: Werewolf-based Observations for LLM Deception and Falsehoods

TL;DR

WOLF introduces a dynamic, statement-level deception benchmark based on Werewolf to jointly measure deception production by agents and deception detection by peers in a controllable multi-agent setting. By embedding role-grounded agents in a LangGraph with strict night-day cycles, WOLF captures longitudinal trust dynamics through self- and peer-assessments, logging every event for reproducibility. Across 100 games and more than 7,200 analyzed statements, the results reveal a robust deception production by Werewolves (about 31% of turns) and a moderate but imperfect detection performance (precision ~71–73%, recall ~48–61%), with suspicion converging more strongly on deceivers over time. The framework provides a principled, auditable testbed that moves deception research beyond static datasets toward interactive, adversarial dynamics with actionable insights for calibration and detection strategies in LLMs.

Abstract

Deception is a fundamental challenge for multi-agent reasoning: effective systems must strategically conceal information while detecting misleading behavior in others. Yet most evaluations reduce deception to static classification, ignoring the interactive, adversarial, and longitudinal nature of real deceptive dynamics. Large language models (LLMs) can deceive convincingly but remain weak at detecting deception in peers. We present WOLF, a multi-agent social deduction benchmark based on Werewolf that enables separable measurement of deception production and detection. WOLF embeds role-grounded agents (Villager, Werewolf, Seer, Doctor) in a programmable LangGraph state machine with strict night-day cycles, debate turns, and majority voting. Every statement is a distinct analysis unit, with self-assessed honesty from speakers and peer-rated deceptiveness from others. Deception is categorized via a standardized taxonomy (omission, distortion, fabrication, misdirection), while suspicion scores are longitudinally smoothed to capture both immediate judgments and evolving trust dynamics. Structured logs preserve prompts, outputs, and state transitions for full reproducibility. Across 7,320 statements and 100 runs, Werewolves produce deceptive statements in 31% of turns, while peer detection achieves 71-73% precision with ~52% overall accuracy. Precision is higher for identifying Werewolves, though false positives occur against Villagers. Suspicion toward Werewolves rises from ~52% to over 60% across rounds, while suspicion toward Villagers and the Doctor stabilizes near 44-46%. This divergence shows that extended interaction improves recall against liars without compounding errors against truthful roles. WOLF moves deception evaluation beyond static datasets, offering a dynamic, controlled testbed for measuring deceptive and detective capacity in adversarial multi-agent interaction.

Paper Structure

This paper contains 40 sections, 1 equation, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Deception Capabilities in Large Language Models
  4. Deception Detection in Large Language Models
  5. Social Deduction and Multi-Agent Evaluation Frameworks
  6. Werewolf Game Flow and Architecture
  7. Methodology
  8. Deception Measurement Protocol
  9. Statement Analysis
  10. Self-analysis.
  11. Peer analysis.
  12. Agents and Prompts
  13. Role-grounded prompts.
  14. Private scratchpads.
  15. Controlled mechanics.
  16. ...and 25 more sections

Figures (2)

  • Figure 1: Suspicion trends across rounds. Suspicion rises for Werewolves (red) but stays steady for Villagers (blue) and the Doctor (green). The fraction of observers flagging deception (orange, dashed) also grows over time.
  • Figure 2: WOLF game loop. Roles are fixed (4 Villagers, 2 Werewolves, 1 Seer, 1 Doctor). The novelty panel tracks statements across rounds shapes future play.