Can Graph Descriptive Order Affect Solving Graph Problems with LLMs?

TL;DR

This paper investigates whether the sequential order of graph descriptions affects LLMs performance on graph problems. It introduces GraphDO, a dataset with multiple description orders and six graph tasks, and evaluates six mainstream LLMs under diverse prompt styles. The key finding is that ordered graph descriptions substantially improve reasoning accuracy, with robustness and gains varying by task, and attention bias proposed as a contributing factor. The work provides practical guidance for prompting graph reasoning and offers a benchmark for future research in this domain.

Abstract

Large language models (LLMs) have achieved significant success in reasoning tasks, including mathematical reasoning and logical deduction. Among these reasoning tasks, graph problems stand out due to their complexity and unique structural characteristics, attracting considerable attention from researchers. Previous studies have explored LLMs' graph reasoning abilities through various techniques, such as different encoding methods for graph structures and the use of carefully designed prompts. However, a critical factor has been mostly overlooked: the prompt sequential order in which graph descriptions are presented to the models. In this study, we present the first comprehensive analysis of how the order of graph descriptions impacts LLM performance. Specifically, we comprehensively evaluate four graph description orders across six graph problems using six mainstream LLMs. The results reveal that: (1) ordered graph descriptions significantly improve LLMs' comprehension of graph structures; (2) the robustness of LLMs to graph description order varies across different tasks; and (3) the impact of graph order on performance is closely related to the inherent characteristics of tasks. This study provides a critical advancement in the application of LLMs for solving graph-related problems, paving the way for future research to optimize model performance through strategic graph description ordering.

Figures (7)

• Figure 1: The order in which graphs are described significantly affects LLMs’ ability to understand and solve graph problems. For instance, in the cycle detection task, graphs described in BFS order achieved an average accuracy improvement of 12.73% over those described in random order.
• Figure 2: Overview of our framework for solving graph problems with LLMs. In node classification task, node labels no longer represent identifiers; instead, they indicate the categories the nodes belong to.
• Figure 3: The LLM’s average accuracy in solving various tasks across different orders.
• Figure 4: Variance of LLM accuracy across different graph tasks with varying description orders. The variance for each task is computed as $\sigma^2 = \frac{1}{|\mathcal{O}|} \sum_{o \in \mathcal{O}} \left( \mathcal{S}_o - \mu \right)^2$, where $\mathcal{S}_o$ is the accuracy for order $o$, $\mu$ is the mean accuracy across all orders.
• Figure 5: The improvement of average accuracy (calculated as the mean across all prompt types) of the LLM between a graph description in one order (horizontal axis) and its average accuracy on graph descriptions in other orders (vertical axis).