Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parametric Knowledge is Not All You Need: Toward Honest Large Language Models via Retrieval of Pretraining Data

Christopher Adrian Kusuma, Muhammad Reza Qorib, Hwee Tou Ng

TL;DR

The paper addresses LLM honesty by defining a knowledge boundary anchored to training data and proposing TIP-TriviaQA, a robust benchmark that reveals when an answer should be withheld. It introduces RETAIN, a Retrieval-Enhanced Inference architecture with a Retriever, an Answerability Classifier, and a Responder that uses retrieved pretraining documents as context to improve both self-knowledge and self-expression. The benchmark combines token-based and vector-based search to automatically identify answerable versus unanswerable questions, enabling fair cross-model evaluation. In experiments with Pythia-12b-deduped, RETAIN outperforms baselines on EM-F1 and PM-F1 and shows strong generalization to HoneSet refusals, highlighting the practical value of retrieval over parametric knowledge for trustworthy LLMs. Overall, the work demonstrates that leveraging pretraining data at inference can enhance honesty and accuracy, offering a concrete path beyond purely parametric knowledge for safer AI systems.

Abstract

Large language models (LLMs) are highly capable of answering questions, but they are often unaware of their own knowledge boundary, i.e., knowing what they know and what they don't know. As a result, they can generate factually incorrect responses on topics they do not have enough knowledge of, commonly known as hallucination. Rather than hallucinating, a language model should be more honest and respond with "I don't know" when it does not have enough knowledge about a topic. Many methods have been proposed to improve LLM honesty, but their evaluations lack robustness, as they do not take into account the knowledge that the LLM has ingested during its pretraining. In this paper, we propose a more robust evaluation benchmark dataset for LLM honesty by utilizing Pythia, a truly open LLM with publicly available pretraining data. In addition, we also propose a novel method for harnessing the pretraining data to build a more honest LLM.

Parametric Knowledge is Not All You Need: Toward Honest Large Language Models via Retrieval of Pretraining Data

TL;DR

The paper addresses LLM honesty by defining a knowledge boundary anchored to training data and proposing TIP-TriviaQA, a robust benchmark that reveals when an answer should be withheld. It introduces RETAIN, a Retrieval-Enhanced Inference architecture with a Retriever, an Answerability Classifier, and a Responder that uses retrieved pretraining documents as context to improve both self-knowledge and self-expression. The benchmark combines token-based and vector-based search to automatically identify answerable versus unanswerable questions, enabling fair cross-model evaluation. In experiments with Pythia-12b-deduped, RETAIN outperforms baselines on EM-F1 and PM-F1 and shows strong generalization to HoneSet refusals, highlighting the practical value of retrieval over parametric knowledge for trustworthy LLMs. Overall, the work demonstrates that leveraging pretraining data at inference can enhance honesty and accuracy, offering a concrete path beyond purely parametric knowledge for safer AI systems.

Abstract

Large language models (LLMs) are highly capable of answering questions, but they are often unaware of their own knowledge boundary, i.e., knowing what they know and what they don't know. As a result, they can generate factually incorrect responses on topics they do not have enough knowledge of, commonly known as hallucination. Rather than hallucinating, a language model should be more honest and respond with "I don't know" when it does not have enough knowledge about a topic. Many methods have been proposed to improve LLM honesty, but their evaluations lack robustness, as they do not take into account the knowledge that the LLM has ingested during its pretraining. In this paper, we propose a more robust evaluation benchmark dataset for LLM honesty by utilizing Pythia, a truly open LLM with publicly available pretraining data. In addition, we also propose a novel method for harnessing the pretraining data to build a more honest LLM.
Paper Structure (37 sections, 1 equation, 5 figures, 16 tables, 1 algorithm)

This paper contains 37 sections, 1 equation, 5 figures, 16 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods to Improve LLMs’ Honesty
  4. LLMs’ Honesty Datasets
  5. Problem Definition
  6. Dataset
  7. Pretraining Data Indexing
  8. Token-Based Search
  9. Vector-Based Search
  10. Final Dataset
  11. Human Annotation
  12. Method
  13. Retriever
  14. Answerability Classifier
  15. Responder
  16. ...and 22 more sections

Figures (5)

  • Figure 1: The dataset creation process consists of two stages of search: token-based and vector-based.
  • Figure 2: The percentage of questions that the models refuse to answer on HoneSet (higher is better).
  • Figure 3: Percentage of correct answers to questions, grouped by the number of documents in the pretraining data that contain all the keywords of each question.
  • Figure 4: Separability of answerable and unanswerable questions.
  • Figure 5: RETAIN's performance when retrieving up to ten pretraining documents.