Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating the Formal Reasoning Capabilities of Large Language Models through Chomsky Hierarchy

Yihong Dong, Xiaoha Jian, Xue Jiang, Xuyuan Guo, Zhiyuan Fan, Jiaru Qian, Kechi Zhang, Jia Li, Zhi Jin, Ge Li

Abstract

The formal reasoning capabilities of LLMs are crucial for advancing automated software engineering. However, existing benchmarks for LLMs lack systematic evaluation based on computation and complexity, leaving a critical gap in understanding their formal reasoning capabilities. Therefore, it is still unknown whether SOTA LLMs can grasp the structured, hierarchical complexity of formal languages as defined by Computation Theory. To address this, we introduce ChomskyBench, a benchmark for systematically evaluating LLMs through the lens of Chomsky Hierarchy. Unlike prior work that uses vectorized classification for neural networks, ChomskyBench is the first to combine full Chomsky Hierarchy coverage, process-trace evaluation via natural language, and deterministic symbolic verifiability. ChomskyBench is composed of a comprehensive suite of language recognition and generation tasks designed to test capabilities at each level. Extensive experiments indicate a clear performance stratification that correlates with the hierarchy's levels of complexity. Our analysis reveals a direct relationship where increasing task difficulty substantially impacts both inference length and performance. Furthermore, we find that while larger models and advanced inference methods offer notable relative gains, they face severe efficiency barriers: achieving practical reliability would require prohibitive computational costs, revealing that current limitations stem from inefficiency rather than absolute capability bounds. A time complexity analysis further indicates that LLMs are significantly less efficient than traditional algorithmic programs for these formal tasks. These results delineate the practical limits of current LLMs, highlight the indispensability of traditional software tools, and provide insights to guide the development of future LLMs with more powerful formal reasoning capabilities.

Evaluating the Formal Reasoning Capabilities of Large Language Models through Chomsky Hierarchy

Abstract

The formal reasoning capabilities of LLMs are crucial for advancing automated software engineering. However, existing benchmarks for LLMs lack systematic evaluation based on computation and complexity, leaving a critical gap in understanding their formal reasoning capabilities. Therefore, it is still unknown whether SOTA LLMs can grasp the structured, hierarchical complexity of formal languages as defined by Computation Theory. To address this, we introduce ChomskyBench, a benchmark for systematically evaluating LLMs through the lens of Chomsky Hierarchy. Unlike prior work that uses vectorized classification for neural networks, ChomskyBench is the first to combine full Chomsky Hierarchy coverage, process-trace evaluation via natural language, and deterministic symbolic verifiability. ChomskyBench is composed of a comprehensive suite of language recognition and generation tasks designed to test capabilities at each level. Extensive experiments indicate a clear performance stratification that correlates with the hierarchy's levels of complexity. Our analysis reveals a direct relationship where increasing task difficulty substantially impacts both inference length and performance. Furthermore, we find that while larger models and advanced inference methods offer notable relative gains, they face severe efficiency barriers: achieving practical reliability would require prohibitive computational costs, revealing that current limitations stem from inefficiency rather than absolute capability bounds. A time complexity analysis further indicates that LLMs are significantly less efficient than traditional algorithmic programs for these formal tasks. These results delineate the practical limits of current LLMs, highlight the indispensability of traditional software tools, and provide insights to guide the development of future LLMs with more powerful formal reasoning capabilities.

Paper Structure

This paper contains 56 sections, 1 equation, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Scope of Formal Reasoning in This Work
  4. LLM Evaluation in Software Engineering
  5. Probing the Reasoning Limits and Theoretical Power of LLMs
  6. Benchmarks for Formal Language and Reasoning
  7. ChomskyBench
  8. Motivation and Foundational Philosophy
  9. Design Principles
  10. Principle 1: Hierarchical Abstraction for Diagnostic Analysis
  11. Principle 2: Task Duality for a Holistic Assessment: Recognition and Generation
  12. Principle 3: Controlled Minimalism to Isolate Core Reasoning
  13. Principle 4: Absolute Algorithmic Verifiability
  14. Detailed Benchmark Architecture and Composition
  15. Systematic Data Generation
  16. ...and 41 more sections

Figures (5)

  • Figure 1: A conceptual comparison between existing benchmarks and our proposed benchmark.
  • Figure 2: Chomsky Hierarchy and ChomskyBench: Four computational complexity levels (Type-3 to Type-0) with corresponding automata models and representative formal languages used for systematic evaluation of LLM formal reasoning capabilities.
  • Figure 3: Construction Process of the ChomskyBench.
  • Figure 4: An Example in ChomskyBench.
  • Figure 5: A multi-faceted visualization of SOTA LLMs for formal reasoning performance on ChomskyBench.