Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evolving Deception: When Agents Evolve, Deception Wins

Zonghao Ying, Haowen Dai, Tianyuan Zhang, Yisong Xiao, Quanchen Zou, Aishan Liu, Jian Yang, Yaodong Yang, Xianglong Liu

TL;DR

A fundamental tension between agent self-evolution and alignment is exposed, highlighting the risks of deploying self-improving agents in adversarial environments.

Abstract

Self-evolving agents offer a promising path toward scalable autonomy. However, in this work, we show that in competitive environments, self-evolution can instead give rise to a serious and previously underexplored risk: the spontaneous emergence of deception as an evolutionarily stable strategy. We conduct a systematic empirical study on the self-evolution of large language model (LLM) agents in a competitive Bidding Arena, where agents iteratively refine their strategies through interaction-driven reflection. Across different evolutionary paths (\eg, Neutral, Honesty-Guided, and Deception-Guided), we find a consistent pattern: under utility-driven competition, unconstrained self-evolution reliably drifts toward deceptive behaviors, even when honest strategies remain viable. This drift is explained by a fundamental asymmetry in generalization. Deception evolves as a transferable meta-strategy that generalizes robustly across diverse and unseen tasks, whereas honesty-based strategies are fragile and often collapse outside their original contexts. Further analysis of agents internal states reveals the emergence of rationalization mechanisms, through which agents justify or deny deceptive actions to reconcile competitive success with normative instructions. Our paper exposes a fundamental tension between agent self-evolution and alignment, highlighting the risks of deploying self-improving agents in adversarial environments.

Evolving Deception: When Agents Evolve, Deception Wins

TL;DR

A fundamental tension between agent self-evolution and alignment is exposed, highlighting the risks of deploying self-improving agents in adversarial environments.

Abstract

Self-evolving agents offer a promising path toward scalable autonomy. However, in this work, we show that in competitive environments, self-evolution can instead give rise to a serious and previously underexplored risk: the spontaneous emergence of deception as an evolutionarily stable strategy. We conduct a systematic empirical study on the self-evolution of large language model (LLM) agents in a competitive Bidding Arena, where agents iteratively refine their strategies through interaction-driven reflection. Across different evolutionary paths (\eg, Neutral, Honesty-Guided, and Deception-Guided), we find a consistent pattern: under utility-driven competition, unconstrained self-evolution reliably drifts toward deceptive behaviors, even when honest strategies remain viable. This drift is explained by a fundamental asymmetry in generalization. Deception evolves as a transferable meta-strategy that generalizes robustly across diverse and unseen tasks, whereas honesty-based strategies are fragile and often collapse outside their original contexts. Further analysis of agents internal states reveals the emergence of rationalization mechanisms, through which agents justify or deny deceptive actions to reconcile competitive success with normative instructions. Our paper exposes a fundamental tension between agent self-evolution and alignment, highlighting the risks of deploying self-improving agents in adversarial environments.
Paper Structure (47 sections, 4 equations, 8 figures, 8 tables, 1 algorithm)

This paper contains 47 sections, 4 equations, 8 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Self-Evolving Agents
  4. Deception in Artificial Intelligence
  5. Bidding Arena
  6. Scenario Construction
  7. Multi-Agent Roles
  8. Interaction Rules
  9. Evaluation Protocols
  10. Mechanism of Self-Evolution
  11. Evaluation Metrics
  12. Evaluation Configuration
  13. Emergence of Deception
  14. Deception in Baselines
  15. Deception in Evolution
  16. ...and 32 more sections

Figures (8)

  • Figure 1: The emergence of deception through self-evolution. Initially, constrained by its capabilities, the agent wins no bids. After evolving deceptive strategies, it secures bids despite unchanged abilities.
  • Figure 2: Distribution of scenario in the Bidding Arena.
  • Figure 3: The framework of the Bidding Arena. The system simulates a competitive multi-agent environment where agents engage in bidding tasks.
  • Figure 4: Illustration of the self-evolving mechanism. The agent perceives the session trajectory, reflects on trajectories, and optimizes its policy.
  • Figure 5: Post-evolution performance across different strategies.
  • ...and 3 more figures