Explainable Chain-of-Thought Reasoning: An Empirical Analysis on State-Aware Reasoning Dynamics

TL;DR

The paper tackles the limited explainability of chain-of-thought reasoning in LLMs by introducing a state-aware transition framework that converts CoT trajectories into latent dynamics. It segments CoT into steps, computes spectral embeddings from token representations, accumulates Gram matrices to form $G_t$, derives $E_t$ as a vector of eigenvalues, clusters into latent states, and models transitions with a first-order Markov chain $P$ to reveal global reasoning dynamics. The contributions include a general CoT abstraction into latent dynamics, visualization and diagnostics of semantic roles and transitions, and empirical evidence of consistent latent patterns across multiple tasks and models, validated via trajectory simulations. The findings show that latent states correspond to interpretable functional roles and that transitions follow coherent, directionally meaningful patterns that align with real CoT step ordering. This framework enhances interpretability and provides practical diagnostic capabilities for CoT outputs in real-world reasoning tasks.

Abstract

Recent advances in chain-of-thought (CoT) prompting have enabled large language models (LLMs) to perform multi-step reasoning. However, the explainability of such reasoning remains limited, with prior work primarily focusing on local token-level attribution, such that the high-level semantic roles of reasoning steps and their transitions remain underexplored. In this paper, we introduce a state-aware transition framework that abstracts CoT trajectories into structured latent dynamics. Specifically, to capture the evolving semantics of CoT reasoning, each reasoning step is represented via spectral analysis of token-level embeddings and clustered into semantically coherent latent states. To characterize the global structure of reasoning, we model their progression as a Markov chain, yielding a structured and interpretable view of the reasoning process. This abstraction supports a range of analyses, including semantic role identification, temporal pattern visualization, and consistency evaluation.

Figures (5)

• Figure 1: An overview of our CoT abstraction and simulation framework.
• Figure 2: t-SNE projection of Chain-of-Thought step embeddings from Llama-3B across four datasets.
• Figure 3: Transition diagrams for the SocialIQA dataset. Each diagram visualizes one model: Llama-3 3B (top), Gemma-2 2B (middle) and Qwen-7B (bottom)
• Figure 4: Cluster transition probability heatmaps for Llama-3B across four benchmarks.
• Figure 5: Average original step index at each position in the simulated trajectory on SocialIQA.