Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GraphDancer: Training LLMs to Explore and Reason over Graphs via Curriculum Reinforcement Learning

Yuyang Bai, Zhuofeng Li, Ping Nie, Jianwen Xie, Yu Zhang

TL;DR

GraphDancer introduces a graph-aware curriculum reinforcement learning framework that trains LLMs to autonomously navigate and reason over heterogeneous graphs via predefined function calls. By interleaving reasoning with graph interactions and progressively increasing task difficulty, a 3B-scale model learns robust multi-hop reasoning and cross-domain generalization, outperforming larger backbones and prompting-based baselines on GRBench. The approach improves the reliability of multi-round interactions and demonstrates transfer to unseen domains and OOD question types, highlighting the potential for grounded, knowledge-intensive AI. While promising, the work also discusses limitations in training stability, domain-dependent transfer, and ethical considerations for real-world graph access and tool use.

Abstract

Large language models (LLMs) increasingly rely on external knowledge to improve factuality, yet many real-world knowledge sources are organized as heterogeneous graphs rather than plain text. Reasoning over such graph-structured knowledge poses two key challenges: (1) navigating structured, schema-defined relations requires precise function calls rather than similarity-based retrieval, and (2) answering complex questions often demands multi-hop evidence aggregation through iterative information seeking. We propose GraphDancer, a reinforcement learning (RL) framework that teaches LLMs to navigate graphs by interleaving reasoning and function execution. To make RL effective for moderate-sized LLMs, we introduce a graph-aware curriculum that schedules training by the structural complexity of information-seeking trajectories using an easy-to-hard biased sampler. We evaluate GraphDancer on a multi-domain benchmark by training on one domain only and testing on unseen domains and out-of-distribution question types. Despite using only a 3B backbone, GraphDancer outperforms baselines equipped with either a 14B backbone or GPT-4o-mini, demonstrating robust cross-domain generalization of graph exploration and reasoning skills. Our code and models can be found at https://yuyangbai.com/graphdancer/ .

GraphDancer: Training LLMs to Explore and Reason over Graphs via Curriculum Reinforcement Learning

TL;DR

GraphDancer introduces a graph-aware curriculum reinforcement learning framework that trains LLMs to autonomously navigate and reason over heterogeneous graphs via predefined function calls. By interleaving reasoning with graph interactions and progressively increasing task difficulty, a 3B-scale model learns robust multi-hop reasoning and cross-domain generalization, outperforming larger backbones and prompting-based baselines on GRBench. The approach improves the reliability of multi-round interactions and demonstrates transfer to unseen domains and OOD question types, highlighting the potential for grounded, knowledge-intensive AI. While promising, the work also discusses limitations in training stability, domain-dependent transfer, and ethical considerations for real-world graph access and tool use.

Abstract

Large language models (LLMs) increasingly rely on external knowledge to improve factuality, yet many real-world knowledge sources are organized as heterogeneous graphs rather than plain text. Reasoning over such graph-structured knowledge poses two key challenges: (1) navigating structured, schema-defined relations requires precise function calls rather than similarity-based retrieval, and (2) answering complex questions often demands multi-hop evidence aggregation through iterative information seeking. We propose GraphDancer, a reinforcement learning (RL) framework that teaches LLMs to navigate graphs by interleaving reasoning and function execution. To make RL effective for moderate-sized LLMs, we introduce a graph-aware curriculum that schedules training by the structural complexity of information-seeking trajectories using an easy-to-hard biased sampler. We evaluate GraphDancer on a multi-domain benchmark by training on one domain only and testing on unseen domains and out-of-distribution question types. Despite using only a 3B backbone, GraphDancer outperforms baselines equipped with either a 14B backbone or GPT-4o-mini, demonstrating robust cross-domain generalization of graph exploration and reasoning skills. Our code and models can be found at https://yuyangbai.com/graphdancer/ .
Paper Structure (23 sections, 5 equations, 3 figures, 9 tables, 1 algorithm)

This paper contains 23 sections, 5 equations, 3 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. The GraphDancer Framework
  3. Problem Setup
  4. LLM Reasoning with Graph Interaction
  5. Training LLMs to Interact with Graphs
  6. Structural Difficulty of Graph Interaction
  7. Graph-Aware Curriculum RL
  8. Training Procedure
  9. Experiments
  10. Experimental Setup
  11. Overall Performance
  12. Impact of Graph-Aware Curriculum
  13. Qualitative Analysis
  14. Error Analysis of Graph Interaction
  15. Related Work
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Overview of GraphDancer framework.
  • Figure 3: Outcome breakdown over all evaluation episodes. Each bar decomposes episodes into Correct and three failure modes: Invalid Format, Loop / Timeout, and Premature Stop. GraphDancer increases the overall success rate and noticeably reduces Loop / Timeout, while the remaining errors are more concentrated in Premature Stop.
  • Figure 4: Original reasoning trajectories for the case study discussed in Section \ref{['sec:case_study']}. The top box shows the Baseline failure, while the bottom box shows the successful trajectory of GraphDancer.