Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Computational Reasoning of Large Language Models

Haitao Wu, Zongbo Han, Joey Tianyi Zhou, Huaxi Huang, Changqing Zhang

TL;DR

This work targets the core capability of computational reasoning in LLMs, proposing a domain-agnostic evaluation framework based on simulating $m$-Tag Turing machines. The introduced Turing Machine Bench (TMBench) measures strict rule-following and internal-state management through multi-step processes, offering self-contained tasks with controllable difficulty and strong theoretical grounding. Empirical results across diverse open-source and proprietary models show clear multi-step performance curves and meaningful correlations with established reasoning benchmarks and real-world tasks, validating TMBench as a cross-domain proxy for reasoning ability. Ablation studies demonstrate the benchmark's scalability and robustness, while also revealing limits of autoregressive models under deep multi-step execution; code and data are publicly available for reproducibility and further exploration.

Abstract

With the rapid development and widespread application of Large Language Models (LLMs), multidimensional evaluation has become increasingly critical. However, current evaluations are often domain-specific and overly complex, limiting their effectiveness as cross-domain proxies for core capabilities. To address these limitations and enable a unified and simple evaluation framework, an ideal proxy task should target a basic capability that generalizes across tasks and is independent of domain-specific knowledge. Turing machine provides a powerful theoretical lens by reducing complex processes to basic, domain-agnostic computational operations. This perspective offers a principled framework for evaluating basic computational abilities essential to a wide range of tasks. Motivated by this abstraction, we introduce \textbf{Turing Machine Bench}, a benchmark designed to assess the ability of LLMs to \textbf{strictly follow rules} and \textbf{accurately manage internal states} for multi-step, referred to as \textbf{computational reasoning}. TMBench incorporates four key features: self-contained and knowledge-agnostic reasoning, a minimalistic multi-step structure, controllable difficulty, and a solid theoretical foundation based on Turing machine. Empirical results demonstrate that TMBench serves as an effective proxy for evaluating computational reasoning on representative LLMs. It produces clear step-wise accuracy curves, revealing LLMs' ability to execute multi-step reasoning processes. By analyzing performance trends across TMBench and established reasoning benchmarks, we find strong correlations with real-world tasks, bridging real-task evaluation with basic ability assessment. These findings suggest that TMBench holds potential as a cross-domain dimension for evaluating reasoning in LLMs. Code and data are available at \href{https://github.com/HaitaoWuTJU/Turing-Machine-Bench}{Repo}.

Computational Reasoning of Large Language Models

TL;DR

This work targets the core capability of computational reasoning in LLMs, proposing a domain-agnostic evaluation framework based on simulating -Tag Turing machines. The introduced Turing Machine Bench (TMBench) measures strict rule-following and internal-state management through multi-step processes, offering self-contained tasks with controllable difficulty and strong theoretical grounding. Empirical results across diverse open-source and proprietary models show clear multi-step performance curves and meaningful correlations with established reasoning benchmarks and real-world tasks, validating TMBench as a cross-domain proxy for reasoning ability. Ablation studies demonstrate the benchmark's scalability and robustness, while also revealing limits of autoregressive models under deep multi-step execution; code and data are publicly available for reproducibility and further exploration.

Abstract

With the rapid development and widespread application of Large Language Models (LLMs), multidimensional evaluation has become increasingly critical. However, current evaluations are often domain-specific and overly complex, limiting their effectiveness as cross-domain proxies for core capabilities. To address these limitations and enable a unified and simple evaluation framework, an ideal proxy task should target a basic capability that generalizes across tasks and is independent of domain-specific knowledge. Turing machine provides a powerful theoretical lens by reducing complex processes to basic, domain-agnostic computational operations. This perspective offers a principled framework for evaluating basic computational abilities essential to a wide range of tasks. Motivated by this abstraction, we introduce \textbf{Turing Machine Bench}, a benchmark designed to assess the ability of LLMs to \textbf{strictly follow rules} and \textbf{accurately manage internal states} for multi-step, referred to as \textbf{computational reasoning}. TMBench incorporates four key features: self-contained and knowledge-agnostic reasoning, a minimalistic multi-step structure, controllable difficulty, and a solid theoretical foundation based on Turing machine. Empirical results demonstrate that TMBench serves as an effective proxy for evaluating computational reasoning on representative LLMs. It produces clear step-wise accuracy curves, revealing LLMs' ability to execute multi-step reasoning processes. By analyzing performance trends across TMBench and established reasoning benchmarks, we find strong correlations with real-world tasks, bridging real-task evaluation with basic ability assessment. These findings suggest that TMBench holds potential as a cross-domain dimension for evaluating reasoning in LLMs. Code and data are available at \href{https://github.com/HaitaoWuTJU/Turing-Machine-Bench}{Repo}.
Paper Structure (35 sections, 28 equations, 5 figures, 2 tables)

This paper contains 35 sections, 28 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Benchmarks
  4. Multi-step Reasoning
  5. Rule System
  6. Turing Machine Bench
  7. Motivation
  8. Turing Machine
  9. Evaluation Metrics
  10. Step Accuracy.
  11. Step-Weighted Accuracy (SWA)
  12. Pass Rate.
  13. Experiments and Results
  14. Experimental Setup
  15. Evaluation of Computational Reasoning
  16. ...and 20 more sections

Figures (5)

  • Figure 1: Illustration of the multi-step performance curve on TMBench across a diverse range of both open-source and proprietary LLMs. Proprietary models demonstrate advantages in computational reasoning abilities, but accuracy still decreases as steps increase.
  • Figure 2: Correlation between TMBench Pass Rate (computational reasoning) and established benchmarks (AIME2024, MATH500, GPQA Diamond and MMLU Pro), where both metrics are min-max normalized, across 12 leading LLMs. Top: Scatter plots with linear fit between TMBench and each benchmark. Bottom: Q-Q plots of regression residuals.
  • Figure 3: Illustration of ablation results. (a) Maximum correct step achieved by Gemini under unbounded-step execution. (b) Impact of decoding temperature on performance. (c) Effect of alphabet types (Roman, Number, Greek, and Special) on model performance. (d) Task difficulty ablation by varying the number of deletions.
  • Figure 4: Illustration of token distrubition.
  • Figure 5: Correlation between TMBench Pass Rate (computational reasoning ability) and Reasoning Score (averaged across AIME2024, MATH500, and GPQA) among LLMs, with a Pearson correlation coefficient of 0.882 with p=1.49e-04, demonstrating the connection between TMBench as an abstract rule-based system simulation and real-world reasoning problem.