The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think

TL;DR

The CoT Encyclopedia introduces a bottom-up, clustering-based framework to analyze and steer long chain-of-thought reasoning in large language models. By extracting diverse classification criteria from model outputs, embedding them semantically, clustering into representative dimensions, and generating contrastive rubrics, it delivers interpretable, task-adaptive analyses and enables practical control over reasoning strategies. Human evaluation confirms improved interpretability over prior methods, and empirical results show that optimal reasoning patterns boost both accuracy and safety, with format (not domain) exerting a larger influence on strategy than content. Furthermore, the framework enables real-time adaptive prompting and smooth strategy interpolation via model merging, offering a practical path to tailoring reasoning to diverse tasks without retraining. These insights advocate format-aware model design and principled reasoning control to enhance reliability and performance in safety-critical applications.

Abstract

Long chain-of-thought (CoT) is an essential ingredient in effective usage of modern large language models, but our understanding of the reasoning strategies underlying these capabilities remains limited. While some prior works have attempted to categorize CoTs using predefined strategy types, such approaches are constrained by human intuition and fail to capture the full diversity of model behaviors. In this work, we introduce the CoT Encyclopedia, a bottom-up framework for analyzing and steering model reasoning. Our method automatically extracts diverse reasoning criteria from model-generated CoTs, embeds them into a semantic space, clusters them into representative categories, and derives contrastive rubrics to interpret reasoning behavior. Human evaluations show that this framework produces more interpretable and comprehensive analyses than existing methods. Moreover, we demonstrate that this understanding enables performance gains: we can predict which strategy a model is likely to use and guide it toward more effective alternatives. Finally, we provide practical insights, such as that training data format (e.g., free-form vs. multiple-choice) has a far greater impact on reasoning behavior than data domain, underscoring the importance of format-aware model design.

Figures (23)

• Figure 1: Comparison between conventional reasoning analysis and the CoT Encyclopedia. Traditional methods use fixed criteria to identify strategies but offer limited guidance for improving reasoning. The CoT Encyclopedia takes a bottom-up approach, uncovering diverse, task-specific strategies and enabling flexible analysis and actionable insights to enhance model performance.
• Figure 2: Overview of the CoT Encyclopedia. The framework constructs a taxonomy of reasoning strategies through five key stages: (1) Classification Criteria Identification – diverse reasoning criteria are identified from model-generated CoTs; (2) Classification Criteria Embedding – these criteria are converted into semantic embeddings; (3) Criteria Compression via Hierarchical Clustering – semantically similar criteria are clustered to form distinct representative categories; (4) Rubric Generation – contrastive rubrics are created to describe and distinguish opposing reasoning patterns within each criterion; (5) Analysis Report Generation – model responses are classified using the rubrics, producing comprehensive reports that interpret their reasoning behaviors. The framework also supports practical use cases such as reasoning pattern analysis and optimal strategy control for performance improvement.
• Figure 3: Human Evaluation Results for CoT Encyclopedia. Human annotators found the generated criteria plausible, their mapping to high-level dimensions sensible, and the overall analysis reasonable.
• Figure 4: Impact of Pattern-Based Instructions on Model Performance. Five approaches are compared: not instructed, unoptimal instructions, random patterns, dataset-wide optimal patterns, and question-specific optimal patterns. Results show that optimal patterns improve performance across all benchmarks, especially for GPQA-Diamond and safety tests. Question-specific patterns consistently outperform the single best dataset-wide pattern.
• Figure 5: Analysis of relationships between question similarity and reasoning strategy similarity across multiple benchmarks. Relationship between question similarity and reasoning strategy similarity. (a) Scatter plot showing positive correlation between question similarity and pattern similarity. (b) Variance analysis showing that pattern similarity becomes more consistent as question similarity increases.