KnowRL: Teaching Language Models to Know What They Know
Sahil Kale, Devendra Singh Dhami
TL;DR
The paper tackles the problem that large language models often misjudge their own capabilities, undermining reliability. It introduces KnowRL, a self-improvement reinforcement learning framework that blends introspection with consensus-based rewarding, leveraging internally generated data to strengthen awareness of feasibility boundaries without external supervision. Through an iterative self-play loop, the approach yields substantial gains in self-knowledge on multiple open-source LLMs, achieving up to 28% intrinsic accuracy and 12% extrinsic F1 improvements, with monotonic progress over cycles. These results suggest a practical path toward safer, more accountable AI in high-stakes domains, with potential extensions to broader reliability and safety challenges across models and languages.
Abstract
Truly reliable AI requires more than simply scaling up knowledge; it demands the ability to know what it knows and when it does not. Yet recent research shows that even the best LLMs misjudge their own competence in more than one in five cases, making any response born of such internal uncertainty impossible to fully trust. Inspired by self-improvement reinforcement learning techniques that require minimal data, we present a simple but powerful framework KnowRL that strengthens a model's internal understanding of its own feasibility boundaries, enabling safer and more responsible behaviour. Our framework combines two components: (i) introspection, where the model generates and classifies tasks it judges feasible or infeasible, and (ii) consensus-based rewarding, where stability of self-knowledge assessment is reinforced through internal agreement. By using internally generated data, this design strengthens consistency in self-knowledge and entirely avoids costly external supervision. In experiments on LLaMA-3.1-8B and Qwen-2.5-7B, KnowRL steadily improved self-knowledge, validated by both intrinsic self-consistency and extrinsic benchmarking. With nothing more than a small seed set and no external supervision, our method drove gains as high as 28% in accuracy and 12% in F1, outperforming baselines in just a few iterations. Our framework essentially unlocks the untapped capacity of LLMs to self-improve their knowledge awareness, opening the door to reliable, more accountable AI and safer deployment in critical applications. Owing to its simplicity and independence from external effort, we encourage applying this reliability-enhancing process to all future models.