Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Are Large-Language Models Graph Algorithmic Reasoners?

Alexander K Taylor, Anthony Cuturrufo, Vishal Yathish, Mingyu Derek Ma, Wei Wang

TL;DR

This work presents MAGMA, the first comprehensive benchmark focused on LLMs completing classical graph algorithms, and provides a critical step toward understanding and improving their structured problem-solving skills.

Abstract

We seek to address a core challenge facing current Large Language Models (LLMs). LLMs have demonstrated superior performance in many tasks, yet continue to struggle with reasoning problems on explicit graphs that require multiple steps. To address this gap, we introduce a novel benchmark designed to evaluate LLM performance on classical algorithmic reasoning tasks on explicit graphs. Our benchmark encompasses five fundamental algorithms: Breadth-First Search (BFS) and Depth-First Search (DFS) for connectivity, Dijkstra's algorithm and Floyd-Warshall algorithm for all nodes shortest path, and Prim's Minimum Spanning Tree (MST-Prim's) algorithm. Through extensive experimentation, we assess the capabilities of state-of-the-art LLMs in executing these algorithms step-by-step and systematically evaluate their performance at each stage. Our findings highlight the persistent challenges LLMs face in this domain and underscore the necessity for advanced prompting techniques and algorithmic instruction to enhance their graph reasoning abilities. This work presents MAGMA, the first comprehensive benchmark focused on LLMs completing classical graph algorithms, and provides a critical step toward understanding and improving their structured problem-solving skills.

Are Large-Language Models Graph Algorithmic Reasoners?

TL;DR

This work presents MAGMA, the first comprehensive benchmark focused on LLMs completing classical graph algorithms, and provides a critical step toward understanding and improving their structured problem-solving skills.

Abstract

We seek to address a core challenge facing current Large Language Models (LLMs). LLMs have demonstrated superior performance in many tasks, yet continue to struggle with reasoning problems on explicit graphs that require multiple steps. To address this gap, we introduce a novel benchmark designed to evaluate LLM performance on classical algorithmic reasoning tasks on explicit graphs. Our benchmark encompasses five fundamental algorithms: Breadth-First Search (BFS) and Depth-First Search (DFS) for connectivity, Dijkstra's algorithm and Floyd-Warshall algorithm for all nodes shortest path, and Prim's Minimum Spanning Tree (MST-Prim's) algorithm. Through extensive experimentation, we assess the capabilities of state-of-the-art LLMs in executing these algorithms step-by-step and systematically evaluate their performance at each stage. Our findings highlight the persistent challenges LLMs face in this domain and underscore the necessity for advanced prompting techniques and algorithmic instruction to enhance their graph reasoning abilities. This work presents MAGMA, the first comprehensive benchmark focused on LLMs completing classical graph algorithms, and provides a critical step toward understanding and improving their structured problem-solving skills.

Paper Structure

This paper contains 68 sections, 13 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Algorithmic Reasoning
  4. Graph Reasoning Tasks with LLMs
  5. Benchmark Construction
  6. Adapting the CLRS Benchmark
  7. Algorithm Selection
  8. Graph Sampling
  9. Problem Trajectories
  10. Prompt Reasoning Strategies
  11. Experimental Settings
  12. Foundation Models
  13. Data splits
  14. Evaluation Metrics
  15. Results
  16. ...and 53 more sections

Figures (13)

  • Figure 1: Illustration of a problem in which only the final step is solved correctly.
  • Figure 2: Illustration of splitting an algorithmic execution step into chat format.
  • Figure 3: Examples of the Input-Output, Intermediate Steps, and Intermediate Steps with Hints prompting strategies.
  • Figure 4: Illustration of the construction of partial algorithm executions evaluation.
  • Figure 5: Average accuracy of GPT-4o models across all graph sizes. Output Only models were instructed to only provide the answer, while Output + Reasoning methods were permitted to use intermediate reasoning.
  • ...and 8 more figures