KnowRL: Exploring Knowledgeable Reinforcement Learning for Factuality

TL;DR

KnowRL tackles the pervasive hallucination problem in slow-thinking LLMs by introducing factuality-supervised reinforcement learning. By integrating a factuality reward derived from external knowledge verification into a GRPO-based training loop, it directly guides the reasoning process to be verifiable and boundary-aware while preserving core reasoning capabilities. Across English and Chinese benchmarks (TruthfulQA, SimpleQA, ChineseSimpleQA, GPQA, AIME 2025) and OlympiadBench tasks, KnowRL reduces hallucinations and increases factual reliability without significant degradation in reasoning performance; ablations demonstrate the central role of the factual reward and the boundary-promoting effect of positive refusal incentives. The work highlights the practical potential of process-oriented supervision for reliable, knowledge-grounded reasoning, while noting efficiency and scalability challenges as directions for future research.

Abstract

Large Language Models (LLMs), particularly slow-thinking models, often exhibit severe hallucination, outputting incorrect content due to an inability to accurately recognize knowledge boundaries during reasoning. While Reinforcement Learning (RL) can enhance complex reasoning abilities, its outcome-oriented reward mechanism often lacks factual supervision over the thinking process, further exacerbating the hallucination problem. To address the high hallucination in slow-thinking models, we propose Knowledge-enhanced RL, KnowRL. KnowRL guides models to perform fact-based slow thinking by integrating a factuality reward, based on knowledge verification, into the RL training process, helping them recognize their knowledge boundaries. KnowRL guides models to perform fact-based slow thinking by integrating a factuality reward, based on knowledge verification, into the RL training process, helping them recognize their knowledge boundaries. This targeted factual input during RL training enables the model to learn and internalize fact-based reasoning strategies. By directly rewarding adherence to facts within the reasoning steps, KnowRL fosters a more reliable thinking process. Experimental results on three hallucination evaluation datasets and two reasoning evaluation datasets demonstrate that KnowRL effectively mitigates hallucinations in slow-thinking models while maintaining their original strong reasoning capabilities. Our code is available at https://github.com/zjunlp/KnowRL.

Figures (6)

• Figure 1: KnowRL reduces hallucinations in slow-thinking models.
• Figure 2: Scaling improves reasoning ability (GPQA) but does not reduce hallucinations (SimpleQA). Results are shown for DeepSeek-R1-Distill models of varying sizes (Qwen-1.5B, Llama-8B, Qwen-14B, Qwen-32B).
• Figure 3: KnowRL framework. We begin by constructing the training dataset and then conduct RL training guided by distinct reward signals.
• Figure 4: KnowRL training dynamics and reasoning behavior analysis. (a)--(b) present the training curves of two model architectures, showing improved factual alignment and stable reasoning length. (c)--(f) illustrate performance trends of Skywork-OR1-7B-Preview across different training steps, including F1 and PAQ (c), reasoning accuracy (d), refusal rate (e), and incorrect rate (f).
• Figure 5: Trends of refusal and incorrect rates on SimpleQA and ChineseSimpleQA when training with the combination of format reward and negative refusal reward across different training steps.