Knowledge Graphs are Implicit Reward Models: Path-Derived Signals Enable Compositional Reasoning

TL;DR

The paper addresses the gap in domain-specific compositional reasoning by grounding model problem solving in axioms encoded as knowledge graphs. It introduces a scalable post-training pipeline (SFT followed by RL) that uses path-derived signals as implicit rewards to encourage intermediate, verifiable reasoning steps. Training on 1-3 hop KG paths enables zero-shot generalization to 4-5 hop queries in the medical domain, outperforming larger frontier models and showing robustness to adversarial perturbations. The work argues that structured, verifiable grounding coupled with carefully designed rewards offers a practical path toward domain-specific intelligent reasoning with broad applicability beyond medicine.

Abstract

Large language models have achieved near-expert performance in structured reasoning domains like mathematics and programming, yet their ability to perform compositional multi-hop reasoning in specialized scientific fields remains limited. We propose a bottom-up learning paradigm in which models are grounded in axiomatic domain facts and compose them to solve complex, unseen tasks. To this end, we present a post-training pipeline, based on a combination of supervised fine-tuning and reinforcement learning (RL), in which knowledge graphs act as implicit reward models. By deriving novel reward signals from knowledge graph paths, we provide verifiable, scalable, and grounded supervision that encourages models to compose intermediate axioms rather than optimize only final answers during RL. We validate this approach in the medical domain, training a 14B model on short-hop reasoning paths (1-3 hops) and evaluating its zero-shot generalization to complex multi-hop queries (4-5 hops). Our experiments show that path-derived rewards act as a "compositional bridge", enabling our model to significantly outperform much larger models and frontier systems like GPT-5.2 and Gemini 3 Pro, on the most difficult reasoning tasks. Furthermore, we demonstrate the robustness of our approach to adversarial perturbations against option-shuffling stress tests. This work suggests that grounding the reasoning process in structured knowledge is a scalable and efficient path toward intelligent reasoning.

Figures (9)

• Figure 1: Compositional Reasoning: A sample 3-hop query that requires systematic traversal of axiomatic triples to make a grounded, multi-step clinical deduction.
• Figure 2: SFT+RL pipeline overview: Schematic of the pipeline from SFT to KG-grounded RL. While SFT enables domain-specific grounding, the path-derived reward signal during RL provides the process supervision necessary for compositional reasoning.
• Figure 3: Accuracy by Hop Length: Our SFT+RL model not only outperforms baselines on 2-3 hop tasks but exhibits a positive performance gradient on unseen 4-, 5-hop reasoning tasks, validating the "compositional bridge" enabled by path-aligned rewards.
• Figure 4: Accuracy by Difficulty Level: Whereas the Base Model’s reasoning collapses as task complexity increases, the SFT+RL pipeline exhibits robustness, maintaining a consistent lead over the SFT-only baseline across all levels.
• Figure 5: Accuracy by ICD-10 Category: Path-aligned rewards consistently improve performance across all 15 medical sub-domains.