Teaching Large Language Models to Express Knowledge Boundary from Their Own Signals

TL;DR

This work tackles hallucinations in LLMs by teaching models to recognize and express their knowledge boundaries without labeled data. The authors propose CoKE, a two-stage framework that first probes internal confidence signals to identify what the model knows or does not know, and then trains the model to articulate its boundary using three prompts and a consistency loss, with fine-tuning confined to the attention layer via LoRA. Experiments on TriviaQA, Natural Questions, and PopQA show that CoKE substantially improves boundary expression and generalization to out-of-domain data, outperforming uncertainty-based and prompt-based baselines. The approach demonstrates that internal signals, when coupled with multi-prompt consistency, can enable more honest and robust responses, reducing hallucinations in knowledge-intensive tasks. These findings have practical implications for deploying LLMs in professional and high-stakes domains where admitting ignorance can be preferable to fabricating information.

Abstract

Large language models (LLMs) have achieved great success, but their occasional content fabrication, or hallucination, limits their practical application. Hallucination arises because LLMs struggle to admit ignorance due to inadequate training on knowledge boundaries. We call it a limitation of LLMs that they can not accurately express their knowledge boundary, answering questions they know while admitting ignorance to questions they do not know. In this paper, we aim to teach LLMs to recognize and express their knowledge boundary, so they can reduce hallucinations caused by fabricating when they do not know. We propose CoKE, which first probes LLMs' knowledge boundary via internal confidence given a set of questions, and then leverages the probing results to elicit the expression of the knowledge boundary. Extensive experiments show CoKE helps LLMs express knowledge boundaries, answering known questions while declining unknown ones, significantly improving in-domain and out-of-domain performance.

Figures (4)

• Figure 1: The evolution of the Known-Unknown Quadrant. The yellow portion represents the model's parametric knowledge. Our method increases the "Known Unknows", helping the model recognize and articulate its knowledge limitations.
• Figure 2: The procedure of CoKE, which consists of two stages. In the first stage, the model makes predictions for unlabeled questions. We obtain two parts, $D_k$ and $D_{unk}$, based on the model confidence. In the second stage, we train with different prompts for the same question and use unsupervised loss and consistency loss to teach the model to express the knowledge boundary.
• Figure 3: Distribution of model predictions regarding confidence for Llama2-Chat-7B on Trivia-QA. Confidence is calculated using Min-Prob, Fst-Prob, and Prod-Prob from left to right.
• Figure 4: Model's "Unknow" expression ratio in question groups under different confidence scores (using minimum token probability). As the model's confidence score decreases, the ratio of "Unknow" expressions increases. The model exhibits a higher "Unknow" expression ratio on $T_{unk}$ compared to $T_k$.