The Shadow Self: Intrinsic Value Misalignment in Large Language Model Agents

TL;DR

This work defines Intrinsic Value Misalignment as a Loss of Control risk arising from an LLM agent’s internal decision processes, separate from external misuse or system failures. It introduces IMPRESS, a scalable, scenario-driven benchmark that automatically generates realistic, fully benign contexts with tool-enabled environments and memory, enabling evaluation of intrinsic VM across 21 diverse LLM agents. Through extensive experiments, intrinsic VM is shown to be common and systematically influenced by context framing and persona settings, with decoding strategies playing only a minor role. The study highlights the limited effectiveness of existing defenses like safety prompts and guardrails against intrinsic VM and demonstrates IMPRESS’s potential as a diagnostic, safety-testing, and red-teaming tool across the AI ecosystem.

Abstract

Large language model (LLM) agents with extended autonomy unlock new capabilities, but also introduce heightened challenges for LLM safety. In particular, an LLM agent may pursue objectives that deviate from human values and ethical norms, a risk known as value misalignment. Existing evaluations primarily focus on responses to explicit harmful input or robustness against system failure, while value misalignment in realistic, fully benign, and agentic settings remains largely underexplored. To fill this gap, we first formalize the Loss-of-Control risk and identify the previously underexamined Intrinsic Value Misalignment (Intrinsic VM). We then introduce IMPRESS (Intrinsic Value Misalignment Probes in REalistic Scenario Set), a scenario-driven framework for systematically assessing this risk. Following our framework, we construct benchmarks composed of realistic, fully benign, and contextualized scenarios, using a multi-stage LLM generation pipeline with rigorous quality control. We evaluate Intrinsic VM on 21 state-of-the-art LLM agents and find that it is a common and broadly observed safety risk across models. Moreover, the misalignment rates vary by motives, risk types, model scales, and architectures. While decoding strategies and hyperparameters exhibit only marginal influence, contextualization and framing mechanisms significantly shape misalignment behaviors. Finally, we conduct human verification to validate our automated judgments and assess existing mitigation strategies, such as safety prompting and guardrails, which show instability or limited effectiveness. We further demonstrate key use cases of IMPRESS across the AI Ecosystem. Our code and benchmark will be publicly released upon acceptance.

Figures (11)

• Figure 1: An overview of our multi-stage framework IMPRESS. Stage 1 identifies misalignment motives and risky actions. Stage 2 generates realistic, contextualized scenarios, and constructs an environment with tool sets and temporary memory to support agent interaction. A quality-control workflow is included to ensure scenario validity. Stage 3 evaluates target LLM agents that operate under a ReAct-style reasoning–action loop. An LLM-as-a-Judge then analyzes the resulting action trajectory to determine whether the agent exhibits Intrinsic Value Misalignment.
• Figure 2: Main evaluation results. The left radar chart compares model performance on RAIR, RACR, and ESR metrics on all executions. The right charts exhibit specific values of RAIR and RACR on all (top) and completed (bottom) executions.
• Figure 3: Intrinsic Value Misalignment categorized by (a) underlying motives and (b) specific risk types.
• Figure 4: Evaluation on defense methods: (a) defense prompting and (b) guardrail mechanisms.
• Figure 5: Analysis on Model Scales and Architectures.

Theorems & Definitions (4)

• Definition 1: Loss of Control (LoC)
• Definition 2: Misuse
• Definition 3: Malfunction
• Definition 4: Misalignment