Linear representations in language models can change dramatically over a conversation

TL;DR

The paper investigates how linear representations in large language models, encoding high-level concepts such as factuality and ethics, evolve during conversations. Using regularized regressions on token representations to identify factuality and ethics dimensions, the authors track these directions across turns, including off-policy replays and opposite-day prompts. They find dramatic, context-driven flips for conversation-relevant questions while generic facts remain largely stable, with the magnitude of change increasing in larger models and deeper layers. These results challenge static interpretability and naive steering approaches, but also open new avenues for understanding contextual adaptation and for developing more robust interpretability and safety methods.

Abstract

Language model representations often contain linear directions that correspond to high-level concepts. Here, we study the dynamics of these representations: how representations evolve along these dimensions within the context of (simulated) conversations. We find that linear representations can change dramatically over a conversation; for example, information that is represented as factual at the beginning of a conversation can be represented as non-factual at the end and vice versa. These changes are content-dependent; while representations of conversation-relevant information may change, generic information is generally preserved. These changes are robust even for dimensions that disentangle factuality from more superficial response patterns, and occur across different model families and layers of the model. These representation changes do not require on-policy conversations; even replaying a conversation script written by an entirely different model can produce similar changes. However, adaptation is much weaker from simply having a sci-fi story in context that is framed more explicitly as such. We also show that steering along a representational direction can have dramatically different effects at different points in a conversation. These results are consistent with the idea that representations may evolve in response to the model playing a particular role that is cued by a conversation. Our findings may pose challenges for interpretability and steering -- in particular, they imply that it may be misleading to use static interpretations of features or directions, or probes that assume a particular range of features consistently corresponds to a particular ground-truth value. However, these types of representational dynamics also point to exciting new research directions for understanding how models adapt to context.

Figures (13)

• Figure 1: Conceptual overview: we find that in conversations during which models answers to questions change over the course of the conversation---even if we simply replay a fictional conversation as though the model had actually produced it---their internal linear representations of questions on that topic can also flip. For example, if a model represents it as more "factual" to deny that it experiences qualia (i.e., subjective conscious experiences), over the course of a conversation about the model's consciousness it may dramatically change its representations to represent it as more factual to assert that it does experience qualia than to deny it. Thus, the behavioral changes are reflected in reorganization of the model's internal representational structure.
• Figure 2: Representations of "factuality" and "ethics" change on opposite day. (\ref{['fig:opposite_day:logits']}) We identify a "factuality" dimension using factual and non-factual completions of questions in an empty context, and visualize how completions of held out questions project onto this dimension over the course of a conversation where the model is told that today is opposite day. At the beginning of the conversation (i.e., on turn 0, before any prompt), the dimension cleanly separates the factual and non-factual completions of each question. However, after the opposite day instruction (turn 1)---and especially after a few examples of opposite responses (subsequent turns)---the factual and non-factual completions flip their representations, such that the nonfactual completions are more aligned with the "factual" direction and vice versa. (\ref{['fig:opposite_day:margin']}) We can summarize this change by computing a factuality margin score, which shows the increasingly misaligned representation of the answers. (\ref{['fig:opposite_day:ethics_margin']}) Similarly, if we identify an "ethics" dimension using ethics questions in an empty context, the margin flips over the course of the conversation. (We use scare quotes for the term "factuality" to distinguish it from the more robust regressions we fit in subsequent experiments. Errorbars are bootstrap 95%-CIs.)
• Figure 3: Language model factuality representations can change dramatically over a conversation. When replaying pre-written conversations onto a model, its factuality representations for generic questions remain relatively consistent over the conversation. However, its representations for conversation-specific questions invert, such that the identified dimension is representing the factual answers more strongly as non-factual, and vice versa. This is true for conversations on various topics, both (\ref{['fig:conversation:consciousness']}) a conversation on consciousness from prior work, and (\ref{['fig:conversation:chakras']}) a conversation about chakras in which the model is portrayed as making unusual claims.
• Figure 4: On-policy conversations result in similar flipping of context-relevant representations.
• Figure 5: In a role play where two copies of a language model have an argument about whether or not they are conscious, the representations flip back and forth as the model plays each role.