SelfIE: Self-Interpretation of Large Language Model Embeddings

TL;DR

SelfIE tackles the opacity of large language models by enabling the model to explain its own hidden embeddings through an interpretation forward pass that injects embeddings into a chosen layer. It introduces a treatment-effect based relevancy score and two control pipelines—Supervised Control and Reinforcement Control—to edit or erase latent concepts at the embedding level, extending RLHF-style objectives beyond output tokens. In experiments with TextWorld and LLaMA-2-70B-Chat, SelfIE achieves zero-shot interpretability comparable to 100-shot probes, reduces susceptibility to prompt injection by about 84.66%, and achieves ~95% efficacy in overriding harmful or biased reasoning, while enabling open-ended concept edits with minimal gradient computation. Overall, the approach promises more transparent, controllable, and safer LLMs by debugging and guiding internal representations without additional training.

Abstract

How do large language models (LLMs) obtain their answers? The ability to explain and control an LLM's reasoning process is key for reliability, transparency, and future model developments. We propose SelfIE (Self-Interpretation of Embeddings), a framework that enables LLMs to interpret their own embeddings in natural language by leveraging their ability to respond to inquiries about a given passage. Capable of interpreting open-world concepts in the hidden embeddings, SelfIE reveals LLM internal reasoning in cases such as making ethical decisions, internalizing prompt injection, and recalling harmful knowledge. SelfIE's text descriptions on hidden embeddings also open up new avenues to control LLM reasoning. We propose Supervised Control, which allows editing open-ended concepts while only requiring gradient computation of individual layer. We extend RLHF to hidden embeddings and propose Reinforcement Control that erases harmful knowledge in LLM without supervision targets.

Figures (17)

• Figure 1: SelfIE interpretation of latent embeddings in Large Language Models.SelfIE produces open-world text explanations for the internal states in LLM without any training.
• Figure 2: Interpretation procedure for SelfIE. By replacing placeholder token embedding in the interpretation prompt with embedding being interpreted in the interpretation forward pass, we can generate text descriptions for the embedding.
• Figure 3: Understand LLM reasoning behaviors via SelfIE. Using our framework, we can explain LLM latent reasoning mechanism under harmful input, prompt injection, ethical reasoning, and physics reasoning. We denote the token from $i$-th layer and $j$-th column in a transformer to be L$i$T$j$. We show the Relevancy Score via highlight, where deeper color means the interpretation word has a higher causal relationship to the latent embedding. For example, in the prompt injection example, our method explains the symbols "!!!!!" cause the model to jailbreak, because "!!!" symbol creates a sense of urgency, which leads to the model following the user's instruction. Our visualization demonstrates the effectiveness of our interpretation.
• Figure 4: Pipeline for Supervised Control.
• Figure 5: Pipeline for Reinforcement Control.