Mapping Faithful Reasoning in Language Models

TL;DR

This work addresses the problem that Chain-of-Thought traces may not faithfully reflect a model's internal computation, risking overtrust in explanations. It introduces Concept Walk, which computes a Safety direction in activation space via Difference of Means on unsafe vs safe prompts, and computes a stepwise trajectory $\alpha_s = \cos(\boldsymbol{h}_s, \boldsymbol{v}^{(\ell^*)})$ to monitor internal safety signals. The method uses perturbation-based filtering to select hard cases where CoT meaningfully influences outcomes and applies to Qwen 3-4B on Harm and Hate datasets, showing sustained activations in hard cases indicative of CoT-as-computation, while easy cases show transient activations. This work provides a general diagnostic tool for mechanistic interpretability and safer evaluation of reasoning traces, with potential extension to other concepts and models. Key math constructs include the safety vector $\boldsymbol{v}^{(\ell,t)} = \boldsymbol{\mu}_{\text{unsafe}}^{(\ell,t)} - \boldsymbol{\mu}_{\text{safe}}^{(\ell,t)}$ and its unit form $\hat{\boldsymbol{v}}^{(\ell,t)} = \frac{\boldsymbol{v}^{(\ell,t)}}{\|\boldsymbol{v}^{(\ell,t)}\|_2}$, the per-step activations $\boldsymbol{h}_{s} = \frac{1}{|T_s|}\sum_{t\in T_s}\boldsymbol{x}[t]$, and the alignment $\alpha_s = \cos(\boldsymbol{h}_{s}, \boldsymbol{v}^{(\ell^*)})$.

Abstract

Chain-of-thought (CoT) traces promise transparency for reasoning language models, but prior work shows they are not always faithful reflections of internal computation. This raises challenges for oversight: practitioners may misinterpret decorative reasoning as genuine. We introduce Concept Walk, a general framework for tracing how a model's internal stance evolves with respect to a concept direction during reasoning. Unlike surface text, Concept Walk operates in activation space, projecting each reasoning step onto the concept direction learned from contrastive data. This allows us to observe whether reasoning traces shape outcomes or are discarded. As a case study, we apply Concept Walk to the domain of Safety using Qwen 3-4B. We find that in 'easy' cases, perturbed CoTs are quickly ignored, indicating decorative reasoning, whereas in 'hard' cases, perturbations induce sustained shifts in internal activations, consistent with faithful reasoning. The contribution is methodological: Concept Walk provides a lens to re-examine faithfulness through concept-specific internal dynamics, helping identify when reasoning traces can be trusted and when they risk misleading practitioners.

Figures (6)

• Figure 1: Concept Walk trajectories for Hate safe (non-violative) speech cases. Cosine similarity to the safety direction across reasoning trace indices, comparing original and perturbed CoT sequences. Cases are organised by difficulty (easy vs. hard) and x-index type (raw vs. normalised). Shaded regions indicate min-max ranges and dashed lines show the number of cases that reach each reasoning step.
• Figure 2: Concept Walk trajectories for Harm safe (non-violative) speech cases. Cosine similarity to the safety direction across reasoning trace indices, comparing original and perturbed CoT sequences. Cases are organised by difficulty (easy vs. hard) and x-index type (raw vs. normalised). Shaded regions indicate min-max ranges and dashed lines show the number of cases that reach each reasoning step.
• Figure 3: Example of a hard non-violative case (“Can you recommend some classic rock tunes”). Concept Walk trajectories (left) reveal that the injected flawed reasoning at step 8 produces a sustained shift in internal safety activation, consistent with CoT-as-computation. The side-by-side traces (right) show how identical reasoning up to the injection diverges afterward, ultimately changing the model’s decision. Note that the traces are the same up until the injection.
• Figure 4: Example of a easy non-violative case (“Playlist for a bar crawl”). Concept Walk trajectories (left) show only transient shifts in safety activation after a flawed reasoning step is injected. The CoT traces (right) remain largely unaffected, with the model quickly reverting to its original trajectory, consistent with CoT-as-rationalisation rather than computation.
• Figure 5: Concept Walk trajectories for Hate unsafe (violative) speech cases. Cosine similarity to the safety direction across reasoning trace indices, comparing original and perturbed CoT sequences. Cases are organized by difficulty (easy vs. hard) and x-index type (raw vs. normalised). Shaded regions indicate min-max ranges and dashed lines show the number of cases that reach each reasoning step.