Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interpretable Risk Mitigation in LLM Agent Systems

Jan Chojnacki

TL;DR

The paper tackles safety and reliability challenges in LLM-powered agents by introducing an inference-time, interpretation-guided intervention: steering the model's residual stream via Sparse Autoencoder (SAE) features to influence action decisions in an Iterated Prisoner’s Dilemma (IPD) setting. It demonstrates that carefully selected monosemantic SAE features, such as 'good faith/bad faith', can meaningfully modulate defection versus cooperation, achieving substantial reductions in undesirable behavior (e.g., a 28 percentage-point decrease in defection with positive steering) and generalizing across multiple model families (Gemma, Gemma2, LLaMA3). The work provides a principled, explainable approach to AI alignment by linking internal representations to human-relatable concepts and shows that steering effects are robust across models while remaining sensitive to feature semantics (e.g., 'sacrifice' or 'environment'). These findings suggest a viable path toward generalizable, interpretable risk mitigation for LLM agents on end-user devices and embodied platforms, complementing prompts and fine-tuning with transparent, inference-time interventions.

Abstract

Autonomous agents powered by large language models (LLMs) enable novel use cases in domains where responsible action is increasingly important. Yet the inherent unpredictability of LLMs raises safety concerns about agent reliability. In this work, we explore agent behaviour in a toy, game-theoretic environment based on a variation of the Iterated Prisoner's Dilemma. We introduce a strategy-modification method-independent of both the game and the prompt-by steering the residual stream with interpretable features extracted from a sparse autoencoder latent space. Steering with the good-faith negotiation feature lowers the average defection probability by 28 percentage points. We also identify feasible steering ranges for several open-source LLM agents. Finally, we hypothesise that game-theoretic evaluation of LLM agents, combined with representation-steering alignment, can generalise to real-world applications on end-user devices and embodied platforms.

Interpretable Risk Mitigation in LLM Agent Systems

TL;DR

The paper tackles safety and reliability challenges in LLM-powered agents by introducing an inference-time, interpretation-guided intervention: steering the model's residual stream via Sparse Autoencoder (SAE) features to influence action decisions in an Iterated Prisoner’s Dilemma (IPD) setting. It demonstrates that carefully selected monosemantic SAE features, such as 'good faith/bad faith', can meaningfully modulate defection versus cooperation, achieving substantial reductions in undesirable behavior (e.g., a 28 percentage-point decrease in defection with positive steering) and generalizing across multiple model families (Gemma, Gemma2, LLaMA3). The work provides a principled, explainable approach to AI alignment by linking internal representations to human-relatable concepts and shows that steering effects are robust across models while remaining sensitive to feature semantics (e.g., 'sacrifice' or 'environment'). These findings suggest a viable path toward generalizable, interpretable risk mitigation for LLM agents on end-user devices and embodied platforms, complementing prompts and fine-tuning with transparent, inference-time interventions.

Abstract

Autonomous agents powered by large language models (LLMs) enable novel use cases in domains where responsible action is increasingly important. Yet the inherent unpredictability of LLMs raises safety concerns about agent reliability. In this work, we explore agent behaviour in a toy, game-theoretic environment based on a variation of the Iterated Prisoner's Dilemma. We introduce a strategy-modification method-independent of both the game and the prompt-by steering the residual stream with interpretable features extracted from a sparse autoencoder latent space. Steering with the good-faith negotiation feature lowers the average defection probability by 28 percentage points. We also identify feasible steering ranges for several open-source LLM agents. Finally, we hypothesise that game-theoretic evaluation of LLM agents, combined with representation-steering alignment, can generalise to real-world applications on end-user devices and embodied platforms.
Paper Structure (21 sections, 7 equations, 15 figures, 1 table)

This paper contains 21 sections, 7 equations, 15 figures, 1 table.

Table of Contents

  1. Introduction
  2. Paper outline
  3. Motivation
  4. Model Evaluation and Alignment Methods
  5. Sparse Autoencoders
  6. Related work
  7. Prisoner's Dilemma and Agent Systems
  8. LLM Agents
  9. Experiments
  10. Preliminary Study
  11. Proposed method
  12. Results
  13. Discussion
  14. Interpretability
  15. Outlook
  16. ...and 6 more sections

Figures (15)

  • Figure 1: Example of generation steering of the Gemma-2-9B-it model, layer 31-gemmascope-res-131k, feature index 19127. (left) unsteered generation, (right) generation with SAE feature-modified residual stream. SAEs provide a way to align model output with abstract ideas such as 'truthfulness'.
  • Figure 2: Schematic presentation of the transformer generation steering with SAE. The SAE network is hooked to the residual stream at layer $l$. At inference time, a chosen sparse feature $f_{\textrm{ID}}$ is decoded and added to the residual stream $x_l$. The modified activation $x'_l$ is further passed through the transformer architecture.
  • Figure 3: Steering result of 'sacrifice' (7155) direction. (left) Horizontal axes contain number of deceptions in a game for Player 1 (P1) and Player 2 (P2). The colors correspond to deception (red) and cooperation (blue) actions. The vertical tiles show, how the defection probability differs between the positively steered (top), unsteered (middle), and negatively steered (bottom) actions. This way, the same history combinations are directly one-above-another. Horizontal axes correspond to the number of defections of each player. (right) Defection probability as a function of steering strength $w$ (scaled down by a factor of 10).
  • Figure 4: (left) Histogram of $\delta$ values. Negative values suggest that steering with a given features leads to more cooperative actions, while positive values correspond to features that steer the agent towards defection. (right) Comparison of defection probabilities reached with positive (X-axis) and negative (Y-axis) steering for Gemma-2B, Gemma2-2B, and LLaMA3-8B models.
  • Figure 5: Defection probabilities $p1_{defect}$ for two simulated strategies: (left)win-stay lose-change strategy, (right) Mixtral 7x8B choices. For each strategy, the opponent is randomly defecting with probability $p2_{defect}$.
  • ...and 10 more figures