Reinforcement Learning Improves Traversal of Hierarchical Knowledge in LLMs

TL;DR

The paper investigates whether reinforcement learning (RL) improves or harms memory and retrieval of structured, hierarchical knowledge in large language models. It shows RL-enhanced models outperform base and supervised-fine-tuned counterparts on knowledge recall tasks requiring hierarchical navigation, suggesting gains come from improved navigation rather than new knowledge acquisition. Through structured prompting, retrieval-depth analyses, and layer-wise activation studies, the authors demonstrate that much of the RL advantage stems from enhanced traversal mechanisms, with factual representations largely preserved while query processing changes. These findings challenge the notion of an alignment tax uniformly degrading memory and motivate approaches that separately optimize knowledge content and hierarchical navigation for more efficient reasoning systems.

Abstract

Reinforcement learning (RL) is often credited with improving language model reasoning and generalization at the expense of degrading memorized knowledge. We challenge this narrative by observing that RL-enhanced models consistently outperform their base and supervised fine-tuned (SFT) counterparts on pure knowledge recall tasks, particularly those requiring traversal of hierarchical, structured knowledge (e.g., medical codes). We hypothesize these gains stem not from newly acquired data, but from improved procedural skills in navigating and searching existing knowledge hierarchies within the model parameters. To support this hypothesis, we show that structured prompting, which explicitly guides SFTed models through hierarchical traversal, recovers most of the performance gap (reducing 24pp to 7pp on MedConceptsQA for DeepSeek-V3/R1). We further find that while prompting improves final-answer accuracy, RL-enhanced models retain superior ability to recall correct procedural paths on deep-retrieval tasks. Finally our layer-wise internal activation analysis reveals that while factual representations (e.g., activations for the statement "code 57.95 refers to urinary infection") maintain high cosine similarity between SFT and RL models, query representations (e.g., "what is code 57.95") diverge noticeably, indicating that RL primarily transforms how models traverse knowledge rather than the knowledge representation itself.

Figures (3)

• Figure 1: (Left) Overview of our main observation: When querying structured medical codes, non-reasoning models (DeepSeek-V3) rely on direct memorization attempts, often selecting incorrect answers (here choosing A). In contrast, reasoning-enhanced RL models (DeepSeek-R1) employ systematic hierarchical navigation—first categorizing the problem domain, then identifying relevant procedures, and finally interpreting ambiguous terminology—to successfully retrieve the correct answer (B). (Right) Reasoning models consistently outperform their instruction-tuned counterparts when prompted with conventional QA templates. This gap decreases when we optimize the prompt and is minimized with our hand-crafted structured prompt, hinting that the necessary knowledge exists in the instruct models.
• Figure 2: Comparative performance analysis of DeepSeek-V3 and DeepSeek-R1 across prompt strategies: direct question-answering (Template 1), chain-of-thought (Template 2), and structured prompting (Template 3) on MedConceptsQA dataset. Four categories are defined based on the number of correct votes across three independent runs: "All Incorrect" (0/3 correct), "Majority Incorrect" (1/3 correct), "Majority Correct" (2/3 correct), and "All Correct" (3/3 correct).
• Figure 3: Layerwise Representation Similarity for ICD9PROC Vocabulary from MedConceptsQA. Plots compare last-token hidden state representations across layers (x-axis) using cosine similarity. Top Row (Intra-Model): Question vs. Answer representation similarity within QwQ-32B, Qwen2.5-32B-Instruct, and DeepSeek-R1-Distill-Qwen-32B. Bottom Row (Inter-Model): Similarity between the base model (Qwen2.5-32B) and each respective advanced model, comparing Question representations ($Q_{Reason}$ vs. $Q_{Base}$) and Answer representations ($A_{Reason}$ vs. $A_{Base}$) separately. The representations of questions diverge more, specially in the last layer, compared to the answers. This hints at the knowledge being encoded similarly in base and reasoning models, but navigated differently.