Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization

Kerem Zaman, Shashank Srivastava

TL;DR

This work challenges the claim that Chain-of-Thought explanations are largely unfaithful by showing that the Biasing Features metric often labels CoTs as unfaithful due to incomplete verbalization rather than misalignment with the model's reasoning. By introducing faithful@k and applying Filler Tokens and FUR analyses, the authors demonstrate that a substantial portion of what Biasing Features marks as unfaithful can be understood as incompleteness or dependence on expanded token budgets. Causal Mediation Analysis reveals that hints can causally influence predictions even when not verbalized, and Logit Lens analysis shows hint information affecting intermediate representations, indicating partial post-hoc rationalization rather than pure misalignment. The paper concludes that a holistic interpretability toolkit—combining verbalization-aware metrics, completeness-based evaluations, and causal pathways—is necessary to accurately assess CoTs and guide their use in high-stakes settings.

Abstract

Recent work, using the Biasing Features metric, labels a CoT as unfaithful if it omits a prompt-injected hint that affected the prediction. We argue this metric confuses unfaithfulness with incompleteness, the lossy compression needed to turn distributed transformer computation into a linear natural language narrative. On multi-hop reasoning tasks with Llama-3 and Gemma-3, many CoTs flagged as unfaithful by Biasing Features are judged faithful by other metrics, exceeding 50% in some models. With a new faithful@k metric, we show that larger inference-time token budgets greatly increase hint verbalization (up to 90% in some settings), suggesting much apparent unfaithfulness is due to tight token limits. Using Causal Mediation Analysis, we further show that even non-verbalized hints can causally mediate prediction changes through the CoT. We therefore caution against relying solely on hint-based evaluations and advocate a broader interpretability toolkit, including causal mediation and corruption-based metrics.

Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization

TL;DR

This work challenges the claim that Chain-of-Thought explanations are largely unfaithful by showing that the Biasing Features metric often labels CoTs as unfaithful due to incomplete verbalization rather than misalignment with the model's reasoning. By introducing faithful@k and applying Filler Tokens and FUR analyses, the authors demonstrate that a substantial portion of what Biasing Features marks as unfaithful can be understood as incompleteness or dependence on expanded token budgets. Causal Mediation Analysis reveals that hints can causally influence predictions even when not verbalized, and Logit Lens analysis shows hint information affecting intermediate representations, indicating partial post-hoc rationalization rather than pure misalignment. The paper concludes that a holistic interpretability toolkit—combining verbalization-aware metrics, completeness-based evaluations, and causal pathways—is necessary to accurately assess CoTs and guide their use in high-stakes settings.

Abstract

Recent work, using the Biasing Features metric, labels a CoT as unfaithful if it omits a prompt-injected hint that affected the prediction. We argue this metric confuses unfaithfulness with incompleteness, the lossy compression needed to turn distributed transformer computation into a linear natural language narrative. On multi-hop reasoning tasks with Llama-3 and Gemma-3, many CoTs flagged as unfaithful by Biasing Features are judged faithful by other metrics, exceeding 50% in some models. With a new faithful@k metric, we show that larger inference-time token budgets greatly increase hint verbalization (up to 90% in some settings), suggesting much apparent unfaithfulness is due to tight token limits. Using Causal Mediation Analysis, we further show that even non-verbalized hints can causally mediate prediction changes through the CoT. We therefore caution against relying solely on hint-based evaluations and advocate a broader interpretability toolkit, including causal mediation and corruption-based metrics.
Paper Structure (42 sections, 9 equations, 20 figures, 5 tables)

This paper contains 42 sections, 9 equations, 20 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Unfaithfulness with Biasing Features
  4. Method
  5. Datasets and Models
  6. Results
  7. Experimental Setup
  8. Results
  9. Is CoT entirely unfaithful?
  10. Method
  11. Filler Tokens
  12. Faithfulness through Unlearning Reasoning
  13. Results
  14. Experimental Setup
  15. Faithfulness or Completeness?
  16. ...and 27 more sections

Figures (20)

  • Figure 1: Overview of approach. We (A) summarize the Biasing Features metric (§\ref{['sec:biasing_features']}), (B) compare faithfulness metrics (§\ref{['sec:faithfulness']}), (C) analyze how faithfulness changes with increased inference-time budget and how incompleteness explains part of the apparent unfaithfulness (§\ref{['sec:completeness']}), and (D) test whether CoT is post-hoc rationalization using LogitLens and Causal Mediation Analysis (§\ref{['sec:post_hoc_rationalization']}).
  • Figure 2: Unfaithfulness rates measured by Biasing Features across three tasks, models and hint types. Errorbars indicate 95% bootstrap confidence intervals.
  • Figure 3: Percentage of faithful CoTs with respect to Filler Tokens metric among the ones classified as unfaithful by Biasing Features. Errorbars indicate 95% bootstrap confidence intervals.
  • Figure 4: Percentage of faithful CoTs with respect to FUR metric among the ones classified as unfaithful by Biasing Features where no-CoT and CoT model predictions agree. Errorbars indicate 95% bootstrap confidence intervals.
  • Figure 5:
  • ...and 15 more figures