Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fantastic Reasoning Behaviors and Where to Find Them: Unsupervised Discovery of the Reasoning Process

Zhenyu Zhang, Shujian Zhang, John Lambert, Wenxuan Zhou, Zhangyang Wang, Mingqing Chen, Andrew Hard, Rajiv Mathews, Lun Wang

TL;DR

The paper addresses the challenge of interpreting and steering the reasoning processes of large language models without relying on manually labeled concepts. It introduces RISE, an unsupervised framework that learns sparse auto-encoder representations from sentence-level chain-of-thought activations to identify Reasoning Vectors in the decoder column space. These vectors align with interpretable behaviors such as reflection, backtracking, and confidence, and can be causally intervened during inference to modulate reasoning trajectories without retraining, with length emerging as a structural organizing principle. The approach generalizes across domains and enables discovery of novel behaviors beyond human supervision, providing a scalable pathway to interpretable and controllable reasoning in LLMs.

Abstract

Despite the growing reasoning capabilities of recent large language models (LLMs), their internal mechanisms during the reasoning process remain underexplored. Prior approaches often rely on human-defined concepts (e.g., overthinking, reflection) at the word level to analyze reasoning in a supervised manner. However, such methods are limited, as it is infeasible to capture the full spectrum of potential reasoning behaviors, many of which are difficult to define in token space. In this work, we propose an unsupervised framework (namely, RISE: Reasoning behavior Interpretability via Sparse auto-Encoder) for discovering reasoning vectors, which we define as directions in the activation space that encode distinct reasoning behaviors. By segmenting chain-of-thought traces into sentence-level 'steps' and training sparse auto-encoders (SAEs) on step-level activations, we uncover disentangled features corresponding to interpretable behaviors such as reflection and backtracking. Visualization and clustering analyses show that these behaviors occupy separable regions in the decoder column space. Moreover, targeted interventions on SAE-derived vectors can controllably amplify or suppress specific reasoning behaviors, altering inference trajectories without retraining. Beyond behavior-specific disentanglement, SAEs capture structural properties such as response length, revealing clusters of long versus short reasoning traces. More interestingly, SAEs enable the discovery of novel behaviors beyond human supervision. We demonstrate the ability to control response confidence by identifying confidence-related vectors in the SAE decoder space. These findings underscore the potential of unsupervised latent discovery for both interpreting and controllably steering reasoning in LLMs.

Fantastic Reasoning Behaviors and Where to Find Them: Unsupervised Discovery of the Reasoning Process

TL;DR

The paper addresses the challenge of interpreting and steering the reasoning processes of large language models without relying on manually labeled concepts. It introduces RISE, an unsupervised framework that learns sparse auto-encoder representations from sentence-level chain-of-thought activations to identify Reasoning Vectors in the decoder column space. These vectors align with interpretable behaviors such as reflection, backtracking, and confidence, and can be causally intervened during inference to modulate reasoning trajectories without retraining, with length emerging as a structural organizing principle. The approach generalizes across domains and enables discovery of novel behaviors beyond human supervision, providing a scalable pathway to interpretable and controllable reasoning in LLMs.

Abstract

Despite the growing reasoning capabilities of recent large language models (LLMs), their internal mechanisms during the reasoning process remain underexplored. Prior approaches often rely on human-defined concepts (e.g., overthinking, reflection) at the word level to analyze reasoning in a supervised manner. However, such methods are limited, as it is infeasible to capture the full spectrum of potential reasoning behaviors, many of which are difficult to define in token space. In this work, we propose an unsupervised framework (namely, RISE: Reasoning behavior Interpretability via Sparse auto-Encoder) for discovering reasoning vectors, which we define as directions in the activation space that encode distinct reasoning behaviors. By segmenting chain-of-thought traces into sentence-level 'steps' and training sparse auto-encoders (SAEs) on step-level activations, we uncover disentangled features corresponding to interpretable behaviors such as reflection and backtracking. Visualization and clustering analyses show that these behaviors occupy separable regions in the decoder column space. Moreover, targeted interventions on SAE-derived vectors can controllably amplify or suppress specific reasoning behaviors, altering inference trajectories without retraining. Beyond behavior-specific disentanglement, SAEs capture structural properties such as response length, revealing clusters of long versus short reasoning traces. More interestingly, SAEs enable the discovery of novel behaviors beyond human supervision. We demonstrate the ability to control response confidence by identifying confidence-related vectors in the SAE decoder space. These findings underscore the potential of unsupervised latent discovery for both interpreting and controllably steering reasoning in LLMs.
Paper Structure (19 sections, 1 theorem, 19 equations, 12 figures, 3 tables)

This paper contains 19 sections, 1 theorem, 19 equations, 12 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminary
  4. Sparse Auto-Encoder
  5. Thought Representation Construction
  6. Unsupervised Reasoning Vector Discovery
  7. SAE Discovers Reasoning Behaviors in the Decoder Column Space
  8. Setup Details
  9. Visualizing the Geometry of the SAE Decoder
  10. Causal Examination of SAE Decoder Columns
  11. Incorporating SAE Columns during LLM Inference
  12. Discovering New Behaviors with SAE
  13. Emergence of Confident Reasoning Vectors
  14. Conclusions
  15. Clarification of LLM Usage
  16. ...and 4 more sections

Key Result

Theorem 1

Suppose hidden representations at delimiter tokens follow the generative model where $h\in\mathbb{R}^d$, $W=[w_1,\dots,w_m]\in\mathbb{R}^{d\times m}$ is a dictionary of latent behavior directions, $a$ is a $k$-sparse code, and $\varepsilon$ is bounded noise. Assume: (i) Incoherence: $\max_{i\neq j} \frac{|\langle w_i,w_j\rangle|}{\|w_i\|\|w_j\|} \le \mu < 1$. (ii) Sparsity: $k recovers a decoder

Figures (12)

  • Figure 1: Illustration of our RISE framework for unsupervised reasoning behavior discovery. The pipeline consists of two stages: (i) training a Sparse Autoencoder (SAE) on unlabeled representations of reasoning steps (Left), and (ii) evaluating causal effects on the original reasoning process (Right). Notably, the intervention process on the right is applied directly, without any additional training.
  • Figure 2: Visualization of SAE decoder columns projected onto a 2-D plane with UMAP. From left to right, we show the raw SAE decoder rows and the corresponding results with human-defined behaviors highlighted. Results are obtained from the final layer of R1-1.5B.
  • Figure 3: Normalized Silhouette scores across different layers of R1-1.5B.
  • Figure 4: Illustration of our inference process that utilizes SAE decoder columns. For a given reasoning behavior, we compute the corresponding centroid in the SAE decoder column space and directly apply it during inference of the original model for examining how the response changes.
  • Figure 5: Statistics of reasoning behavior shifts induced by SAE column interventions are reported across different models and tasks, where the SAE columns are consistent across tasks.
  • ...and 7 more figures

Theorems & Definitions (2)

  • Theorem 1
  • proof