Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TuringQ: Benchmarking AI Comprehension in Theory of Computation

Pardis Sadat Zahraei, Ehsaneddin Asgari

TL;DR

TuringQ is presented, the first benchmark designed to evaluate the reasoning capabilities of large language models (LLMs) in the theory of computation, and an automated LLM-based evaluation system is proposed that demonstrates competitive accuracy when compared to human evaluation.

Abstract

We present TuringQ, the first benchmark designed to evaluate the reasoning capabilities of large language models (LLMs) in the theory of computation. TuringQ consists of 4,006 undergraduate and graduate-level question-answer pairs, categorized into four difficulty levels and covering seven core theoretical areas. We evaluate several open-source LLMs, as well as GPT-4, using Chain of Thought prompting and expert human assessment. Additionally, we propose an automated LLM-based evaluation system that demonstrates competitive accuracy when compared to human evaluation. Fine-tuning a Llama3-8B model on TuringQ shows measurable improvements in reasoning ability and out-of-domain tasks such as algebra. TuringQ serves as both a benchmark and a resource for enhancing LLM performance in complex computational reasoning tasks. Our analysis offers insights into LLM capabilities and advances in AI comprehension of theoretical computer science.

TuringQ: Benchmarking AI Comprehension in Theory of Computation

TL;DR

TuringQ is presented, the first benchmark designed to evaluate the reasoning capabilities of large language models (LLMs) in the theory of computation, and an automated LLM-based evaluation system is proposed that demonstrates competitive accuracy when compared to human evaluation.

Abstract

We present TuringQ, the first benchmark designed to evaluate the reasoning capabilities of large language models (LLMs) in the theory of computation. TuringQ consists of 4,006 undergraduate and graduate-level question-answer pairs, categorized into four difficulty levels and covering seven core theoretical areas. We evaluate several open-source LLMs, as well as GPT-4, using Chain of Thought prompting and expert human assessment. Additionally, we propose an automated LLM-based evaluation system that demonstrates competitive accuracy when compared to human evaluation. Fine-tuning a Llama3-8B model on TuringQ shows measurable improvements in reasoning ability and out-of-domain tasks such as algebra. TuringQ serves as both a benchmark and a resource for enhancing LLM performance in complex computational reasoning tasks. Our analysis offers insights into LLM capabilities and advances in AI comprehension of theoretical computer science.

Paper Structure

This paper contains 25 sections, 2 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluating Computational Reasoning Capabilities of LLMs
  4. Automated LLM Evaluation
  5. The TuringQ Dataset
  6. Data Collection
  7. Question Type Distribution
  8. Experiments
  9. Evaluation Metrics
  10. AI-Driven Assessment
  11. Model Specialization
  12. Results
  13. Performance Evaluation
  14. Category-Specific Performance Analysis
  15. Impact of Difficulty Levels on Model Performance
  16. ...and 10 more sections

Figures (2)

  • Figure 1: TuringQ Dataset and its Evaluation Framework. This diagram presents the TuringQ dataset, a comprehensive resource for theory of computation, and illustrates the automated assessment of LLMs using Llama3-8B. It showcases sample questions, LLM responses, and their evaluations by the AI evaluator. The fine-tuned Llama3-8B-ft-TuringQ model demonstrates improved performance but still encounters certain challenges in addressing TuringQ questions.
  • Figure 2: Category and Difficulty Level Distribution in the TuringQ Dataset