Position: Explaining Behavioral Shifts in Large Language Models Requires a Comparative Approach

TL;DR

The paper tackles the problem of explaining behavioral shifts that occur in large language models after interventions such as scaling, fine-tuning, or prompting. It proposes a formal Comparative XAI (Δ-XAI) framework that compares model behavior across consecutive checkpoints and defines how to attribute shifts to underlying internal changes using paired explanations. It identifies a set of desiderata (comparability, validity, actionability, monitoring) and maps six families of comparative explainers to these goals, illustrating the approach with an end-to-end pipeline and a concrete experiment that localizes and mitigates a harmful shift. The work provides a governance-oriented view on model updates, advocating for transition-aware reporting, robust explanations, and falsifiable causal tests to support safer and more accountable deployment of LLMs.

Abstract

Large-scale foundation models exhibit behavioral shifts: intervention-induced behavioral changes that appear after scaling, fine-tuning, reinforcement learning or in-context learning. While investigating these phenomena have recently received attention, explaining their appearance is still overlooked. Classic explainable AI (XAI) methods can surface failures at a single checkpoint of a model, but they are structurally ill-suited to justify what changed internally across different checkpoints and which explanatory claims are warranted about that change. We take the position that behavioral shifts should be explained comparatively: the core target should be the intervention-induced shift between a reference model and an intervened model, rather than any single model in isolation. To this aim we formulate a Comparative XAI ($Δ$-XAI) framework with a set of desiderata to be taken into account when designing proper explaining methods. To highlight how $Δ$-XAI methods work, we introduce a set of possible pipelines, relate them to the desiderata, and provide a concrete $Δ$-XAI experiment.

Figures (6)

• Figure 1: Comparison between a classical XAI framework (left) and the $\Delta$-XAI framework (right). An XAI explainer produces an explanation for a single model and the explanations of successive checkpoints are independent. In the $\Delta$-XAI framework, each checkpoint is associated with a behavior measured by a metric $B$. There is a behavioral shift when the change $\Delta B$ between consecutive checkpoints exceeds a task-dependent threshold, thereby defining $M_{\text{pre}}\xspace$ and $M_{\text{post}}\xspace$. An explainer $\Phi$ is applied to both checkpoints under matched conditions, and a comparative explainer $\Phi_\Delta$ maps the resulting explanations to a comparative explanation $e_\Delta$.
• Figure 2: Taxonomy of $\Delta$-XAI desiderata, organized by category and by whether they pertain to the comparative explainer $\Phi_{\Delta}$ or to the resulting comparative explanation $e_{\Delta}$.
• Figure 3: Bipartite graph mapping comparative explainers to the desiderata they most naturally support.
• Figure 4: End-to-end $\Delta$-XAI pipeline with desiderata coverage.Setup: observe $\Delta B$ on $X_\Delta$ under metric $B$ and define an onset window around $M_{\bar{t}}$; Stage 1--2: probing yields $e^{\text{probe}}_\Delta$ to localize onset and nominate candidates (D1, D2, D4, D8), then mechanistic interventions yield $e^{\text{mech}}_\Delta$ as bidirectional causal evidence and actionable components (D4, D7); plus robustness tests for D5--D6).
• Figure 6: Comparative analysis of hidden representations in $M_{\text{pre}}$ and its unsafer variant $M_{\text{post}}$.(a) Linear CKA similarity between $M_{\text{pre}}$ and $M_{\text{post}}$ hidden representations across transformer layers (averaged over prompts), localizing the main representational divergence to late layers, and particularly to the third-to-last layer (D1, D2). (b) Box plots with mean shown as a triangle suggest that third-to-last layer activation patching increases similarity of $M_{\text{post}}$ outputs to $M_{\text{pre}}$ across prompts (D6). (c) Activation steering at the same layer using a probe-derived direction moves outputs (here, we show the first two sentences) toward $M_{\text{pre}}$-like advice (D6, D7).

Theorems & Definitions (3)

• Example 1
• Example 2
• Example 3