Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Graph Reasoning Paradigm: Structured and Symbolic Reasoning with Topology-Aware Reinforcement Learning for Large Language Models

Runxuan Liu, Xianhao Ou, Xinyan Ma, Jiyuan Wang, Jiafeng Liang, Jiaqi Li, Tao He, Zheng Chu, Rongchuan Mu, Zekun Wang, Baoxin Wang, Dayong Wu, Ming Liu, Shijin Wang, Guoping Hu, Bing Qin

TL;DR

The paper tackles the bottleneck of semantic evaluation in plain-text LLM reasoning by introducing Graph Reasoning Paradigm (GRP), which codifies reasoning as graph-structured, step-labeled reasoning traces. Built on GRP, the authors develop PASC-GRPO, a topology-aware reinforcement learning framework with stratified clipping to optimize reasoning quality while mitigating reward hacking, using graph-based outcome rewards. Empirical results on mathematical reasoning and code generation show significant accuracy gains over strong baselines, with reduced reasoning length and improved efficiency, including competitive performance against larger or more resource-intensive models. The work demonstrates the value of structured symbolic reasoning and topology-aware supervision for more reliable and scalable LLM reasoning, with plans to release data, models, and code later.

Abstract

Long Chain-of-Thought (LCoT), achieved by Reinforcement Learning with Verifiable Rewards (RLVR), has proven effective in enhancing the reasoning capabilities of Large Language Models (LLMs). However, reasoning in current LLMs is primarily generated as plain text, where performing semantic evaluation on such unstructured data creates a computational bottleneck during training. Despite RLVR-based optimization, existing methods still suffer from coarse-grained supervision, reward hacking, high training costs, and poor generalization. To address these issues, we propose the Graph Reasoning Paradigm (GRP), which realizes structured and symbolic reasoning, implemented via graph-structured representations with step-level cognitive labels. Building upon GRP, we further design Process-Aware Stratified Clipping Group Relative Policy Optimization (PASC-GRPO), which leverages structured evaluation to replace semantic evaluation, achieves process-aware verification through graph-structured outcome rewards, and mitigates reward hacking via stratified clipping advantage estimation. Experiments demonstrate significant improvements across mathematical reasoning and code generation tasks. Data, models, and code will be released later.

Graph Reasoning Paradigm: Structured and Symbolic Reasoning with Topology-Aware Reinforcement Learning for Large Language Models

TL;DR

The paper tackles the bottleneck of semantic evaluation in plain-text LLM reasoning by introducing Graph Reasoning Paradigm (GRP), which codifies reasoning as graph-structured, step-labeled reasoning traces. Built on GRP, the authors develop PASC-GRPO, a topology-aware reinforcement learning framework with stratified clipping to optimize reasoning quality while mitigating reward hacking, using graph-based outcome rewards. Empirical results on mathematical reasoning and code generation show significant accuracy gains over strong baselines, with reduced reasoning length and improved efficiency, including competitive performance against larger or more resource-intensive models. The work demonstrates the value of structured symbolic reasoning and topology-aware supervision for more reliable and scalable LLM reasoning, with plans to release data, models, and code later.

Abstract

Long Chain-of-Thought (LCoT), achieved by Reinforcement Learning with Verifiable Rewards (RLVR), has proven effective in enhancing the reasoning capabilities of Large Language Models (LLMs). However, reasoning in current LLMs is primarily generated as plain text, where performing semantic evaluation on such unstructured data creates a computational bottleneck during training. Despite RLVR-based optimization, existing methods still suffer from coarse-grained supervision, reward hacking, high training costs, and poor generalization. To address these issues, we propose the Graph Reasoning Paradigm (GRP), which realizes structured and symbolic reasoning, implemented via graph-structured representations with step-level cognitive labels. Building upon GRP, we further design Process-Aware Stratified Clipping Group Relative Policy Optimization (PASC-GRPO), which leverages structured evaluation to replace semantic evaluation, achieves process-aware verification through graph-structured outcome rewards, and mitigates reward hacking via stratified clipping advantage estimation. Experiments demonstrate significant improvements across mathematical reasoning and code generation tasks. Data, models, and code will be released later.
Paper Structure (56 sections, 14 equations, 13 figures, 4 tables)

This paper contains 56 sections, 14 equations, 13 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Structured cognition.
  4. RL-enhanced Reasoning.
  5. Methodology
  6. Graph Reasoning Paradigm
  7. Step-level Cognitive Labels
  8. Node-based Reasoning Representation
  9. Data Synthesis and SFT
  10. Generating and Validating Reasoning Traces.
  11. Graph-Structured Translation.
  12. Graphical CoT Verification and Refinement.
  13. Supervised Fine-Tuning with Mixed Reasoning Formats.
  14. PASC-GRPO
  15. Process-Aware Graph Rewards
  16. ...and 41 more sections

Figures (13)

  • Figure 1: Comparison between traditional Long Chain-of-Thought and Graph Reasoning Paradigm.
  • Figure 2: Overview of the proposed method.
  • Figure 3: Illustration of the Stratified Clipping Advantage Estimation.
  • Figure 4: (Top) Accuracy comparison between PASC-GRPO and its variants without connectivity and ERS Ratio rewards on AMC23. (Bottom) Corresponding changes in mean reasoning length during training.
  • Figure 5: Step-level Cognitive Labels.
  • ...and 8 more figures