The Shape of Reasoning: Topological Analysis of Reasoning Traces in Large Language Models

TL;DR

This work introduces a topological data analysis framework to evaluate the shape of reasoning traces produced by large language models. By generating traces, aligning them to expert steps, and extracting Vietoris–Rips persistence-based features, the approach captures high-dimensional geometric structure that outperforms traditional graph metrics in predicting reasoning quality as proxied by Smith–Waterman alignment. The study demonstrates that a compact set of stable TDA features correlates with trace quality and can serve as a practical signal for reinforcement learning and monitoring to reduce reliance on costly human ratings. The findings suggest that effective reasoning manifests as coherent, multi-scale geometric patterns rather than purely relational graphs, offering a path toward label-efficient, topology-guided improvements in LLM reasoning. Practical impact includes improved automated evaluation of reasoning traces and potential topology-informed reward signals for training more reliable, faithful reasoning in LLMs.

Abstract

Evaluating the quality of reasoning traces from large language models remains understudied, labor-intensive, and unreliable: current practice relies on expert rubrics, manual annotation, and slow pairwise judgments. Automated efforts are dominated by graph-based proxies that quantify structural connectivity but do not clarify what constitutes high-quality reasoning; such abstractions can be overly simplistic for inherently complex processes. We introduce a topological data analysis (TDA)-based evaluation framework that captures the geometry of reasoning traces and enables label-efficient, automated assessment. In our empirical study, topological features yield substantially higher predictive power for assessing reasoning quality than standard graph metrics, suggesting that effective reasoning is better captured by higher-dimensional geometric structures rather than purely relational graphs. We further show that a compact, stable set of topological features reliably indicates trace quality, offering a practical signal for future reinforcement learning algorithms.

Figures (7)

• Figure 1: Visualization of Smith-Waterman alignment diagram. Please refer to Appendix \ref{['app:swad']} for the real example.
• Figure 2: Silhouette plot based on TDA feature correlations across all models.
• Figure 3: Correlation Heatmap for Qwen3-32B.
• Figure 4: Correlation Heatmap for Deepseek-r1-32B.
• Figure 5: Correlation Heatmap for GPT-OSS-20B.