Structure-R1: Dynamically Leveraging Structural Knowledge in LLM Reasoning through Reinforcement Learning

TL;DR

This paper introduces Structure-R1, a reinforcement learning–driven framework that transforms retrieved text into dynamic, structured representations to enhance reasoning in knowledge-intensive tasks. By combining format-aware prompting, a warm-up supervision phase, and a self-reward structural verification mechanism, Structure-R1 learns a content representation policy that can generate novel formats tailored to each query. The authors provide a theoretical analysis based on information density to justify why adaptive structures improve reasoning and demonstrate state-of-the-art performance on seven benchmarks using a 7B-scale backbone, often matching larger models. The work offers practical gains for structure-aware RAG systems and contributes open-source code and data for reproducibility.

Abstract

Large language models (LLMs) have demonstrated remarkable advances in reasoning capabilities. However, their performance remains constrained by limited access to explicit and structured domain knowledge. Retrieval-Augmented Generation (RAG) addresses this by incorporating external information as context to augment reasoning. Nevertheless, traditional RAG systems typically operate over unstructured and fragmented text, resulting in low information density and suboptimal reasoning. To overcome these limitations, we propose \textsc{Structure-R1}, a novel framework that transforms retrieved content into structured representations optimized for reasoning. Leveraging reinforcement learning, \textsc{Structure-R1} learns a content representation policy that dynamically generates and adapts structural formats based on the demands of multi-step reasoning. Unlike prior methods that rely on fixed schemas, our approach adopts a generative paradigm capable of producing task-specific structures tailored to individual queries. To ensure the quality and reliability of these representations, we introduce a self-reward structural verification mechanism that checks whether the generated structures are both correct and self-contained. Extensive experiments on seven knowledge-intensive benchmarks show that \textsc{Structure-R1} consistently achieves competitive performance with a 7B-scale backbone model and matches the performance of much larger models. Additionally, our theoretical analysis demonstrates how structured representations enhance reasoning by improving information density and contextual clarity. Our code and data are available at: https://github.com/jlwu002/sr1.

Figures (5)

• Figure 1: Scattered information across documents can hinder reasoning. Structuring content into formats like tables or timelines helps consolidate key details and improve accuracy.
• Figure 2: Reward calculation for GRPO training. The process evaluates the correctness of the generated answers and ensures that the extracted structures are self-contained, coherent, and informative.
• Figure 3: Sensitivity analysis of model performance with respect to different values of $\lambda$. By averaging results across all datasets, we observe that the model achieves the most significant performance gain when $\lambda = 0.2$. Results are shown for HotpotQA (top left), 2Wiki. (top right), PopQA (bottom left), and Bamboogle (bottom right).
• Figure 4: Prompt template used in our Structure-R1 framework.
• Figure 5: R1-like prompt template used in our Structure-R1 framework and the R1 baseline.

Theorems & Definitions (4)

• Definition 1: Retrieved-context Reasoning Task
• Definition 2: Structure Enhanced QA Task
• Theorem 1: Self-Defined Structures Fit Better
• Theorem 2: Optimal Structure Means Minimal Error