Schoenfeld's Anatomy of Mathematical Reasoning by Language Models

TL;DR

The paper introduces ThinkARM, an automatic, sentence-level annotation framework grounded in Schoenfeld's Episode Theory to dissect large-language-model reasoning traces in mathematical problem solving. By analyzing a large, diverse set of traces from Omni-MATH across 15 models, it reveals reproducible episode-level dynamics, including a three-phase cognitive heartbeat and distinct transitions that separate reasoning from non-reasoning behavior. Two diagnostic case studies show that exploration serves as a critical branching point for correctness and that efficiency techniques reshape episode distributions in qualitatively different ways, not just reduce length. The work demonstrates that episode-level representations enable systematic analysis of how reasoning is structured, stabilized, and altered in modern language models, with implications for evaluation and design of more predictable reasoning processes.

Abstract

Large language models increasingly expose reasoning traces, yet their underlying cognitive structure and steps remain difficult to identify and analyze beyond surface-level statistics. We adopt Schoenfeld's Episode Theory as an inductive, intermediate-scale lens and introduce ThinkARM (Anatomy of Reasoning in Models), a scalable framework that explicitly abstracts reasoning traces into functional reasoning steps such as Analysis, Explore, Implement, Verify, etc. When applied to mathematical problem solving by diverse models, this abstraction reveals reproducible thinking dynamics and structural differences between reasoning and non-reasoning models, which are not apparent from token-level views. We further present two diagnostic case studies showing that exploration functions as a critical branching step associated with correctness, and that efficiency-oriented methods selectively suppress evaluative feedback steps rather than uniformly shortening responses. Together, our results demonstrate that episode-level representations make reasoning steps explicit, enabling systematic analysis of how reasoning is structured, stabilized, and altered in modern language models.

Figures (5)

• Figure 1: A condensed example of a reasoning trace that is annotated in our framework. Each sentence in response is tagged with one of the eight episode categories.
• Figure 2: Word clouds visualizing the most frequent semantic tokens for each cognitive episode. The distinct lexical distributions highlight the semantically separable cognitive patterns captured by ThinkARM.
• Figure 3: Thinking dynamics of cognitive episodes reveal a three-phase "heartbeat" pattern of reasoning models: (1) Initialization, dominated by Read, Analyze, and Plan; (2) Execution, where Implement peaks; and (3) Convergence, characterized by a surge in Verify and Monitor before the final Answer.
• Figure 4: The ThinkARM Framework. For each question-response pair, the model response is first segmented into sentences. They are then tagged by the annotation models in batches, along with information about the guidebook, question, context, and format. The guidebook is in Appendix \ref{['sec:refined_annotation_guidebook']}, the prompt template is in Appendix \ref{['sec:annotation_prompt']}.
• Figure 5: The prompt template we used to annotate the reasoning episode.