Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating the Systematic Reasoning Abilities of Large Language Models through Graph Coloring

Alex Heyman, Joel Zylberberg

TL;DR

This study uses graph $k$-coloring as a controlled benchmark to probe the systematic reasoning capabilities of large language models (LLMs) and large reasoning models (LRMs). It benchmarks six models across five small-graph problem sets ($4 \le n \le 8$, $2 \le k \le 4$) and four semantic frames, incorporating difficulty metrics like greedy score $g$ and uncolorability types to assess problem hardness. Results show standard LLMs struggle with even simple instances (often $>60\%$ error in harder cases), while LRMs substantially reduce errors but do not reach perfect accuracy, particularly as problems demand more extensive possibility-space exploration. The findings highlight meaningful framing effects, substantial progress in LLM reasoning but persistent reliability limits, and significant computational costs, underscoring the need for architectural and training innovations alongside robust benchmarking to guide future AI development.

Abstract

Contemporary large language models are powerful problem-solving tools, but they exhibit weaknesses in their reasoning abilities which ongoing research seeks to mitigate. We investigate graph coloring as a means of evaluating an LLM's capacities for systematic step-by-step reasoning and possibility space exploration, as well as effects of semantic problem framing. We test Claude 3.5 Sonnet, Llama 3.1 405B, Gemini 1.5 Pro, GPT-4o, o1-mini, and DeepSeek-R1 on a dataset of $k$-coloring problems with $2 \leq k \leq 4$ and vertex count $4 \leq n \leq 8$, using partial algorithmic solvers to further categorize problems by difficulty. In addition to substantial but varying framing effects, we find that all models except o1-mini and R1 exhibit $>60\%$ error rates on difficult problem types in all frames ($>15\%$ for o1-mini and $>10\%$ for R1), and no model achieves perfect accuracy even in the simple domain of 2-coloring 4-vertex graphs. Our results highlight both the considerable recent progress in LLM systematic reasoning and the limits of its reliability, especially in relation to increasing computational costs. We expect that more complex graph coloring problems, and procedural generation of arbitrary-complexity reasoning problems more broadly, offer further untapped potential for LLM benchmarking.

Evaluating the Systematic Reasoning Abilities of Large Language Models through Graph Coloring

TL;DR

This study uses graph -coloring as a controlled benchmark to probe the systematic reasoning capabilities of large language models (LLMs) and large reasoning models (LRMs). It benchmarks six models across five small-graph problem sets (, ) and four semantic frames, incorporating difficulty metrics like greedy score and uncolorability types to assess problem hardness. Results show standard LLMs struggle with even simple instances (often error in harder cases), while LRMs substantially reduce errors but do not reach perfect accuracy, particularly as problems demand more extensive possibility-space exploration. The findings highlight meaningful framing effects, substantial progress in LLM reasoning but persistent reliability limits, and significant computational costs, underscoring the need for architectural and training innovations alongside robust benchmarking to guide future AI development.

Abstract

Contemporary large language models are powerful problem-solving tools, but they exhibit weaknesses in their reasoning abilities which ongoing research seeks to mitigate. We investigate graph coloring as a means of evaluating an LLM's capacities for systematic step-by-step reasoning and possibility space exploration, as well as effects of semantic problem framing. We test Claude 3.5 Sonnet, Llama 3.1 405B, Gemini 1.5 Pro, GPT-4o, o1-mini, and DeepSeek-R1 on a dataset of -coloring problems with and vertex count , using partial algorithmic solvers to further categorize problems by difficulty. In addition to substantial but varying framing effects, we find that all models except o1-mini and R1 exhibit error rates on difficult problem types in all frames ( for o1-mini and for R1), and no model achieves perfect accuracy even in the simple domain of 2-coloring 4-vertex graphs. Our results highlight both the considerable recent progress in LLM systematic reasoning and the limits of its reliability, especially in relation to increasing computational costs. We expect that more complex graph coloring problems, and procedural generation of arbitrary-complexity reasoning problems more broadly, offer further untapped potential for LLM benchmarking.

Paper Structure

This paper contains 24 sections, 36 figures.

Table of Contents

  1. Introduction
  2. Graph Coloring and LLM Reasoning
  3. Methods
  4. Problems
  5. Difficulty Categorization
  6. Frames
  7. Models
  8. Prompting and Response Evaluation
  9. Code and Data
  10. Results
  11. Standard LLMs
  12. LRMs -- o1-mini and DeepSeek-R1
  13. Discussion
  14. Greedy Score Algorithm
  15. Example Prompts
  16. ...and 9 more sections

Figures (36)

  • Figure 1: Left: A simple undirected graph with 4 vertices and 4 edges. Center: A valid 2-coloring of the graph. Right: A different 4-vertex, 4-edge graph that cannot be 2-colored.
  • Figure 2: Error rate for each standard LLM, problem set, problem type, and frame. The lighter-colored segment of each bar is the space between the lowest-error and highest-error frames.
  • Figure 3: Error rate for each LRM, edge-selected problem set, problem type, and frame. The lighter-colored segment of each bar is the space between the errors of the two tested frames. The y-axis is compressed in comparison to Figure \ref{['fig:all_complete_models']} to better show the differences between the smaller error rates.
  • Figure :
  • Figure :
  • ...and 31 more figures