Estimating Knowledge in Large Language Models Without Generating a Single Token

TL;DR

KEEN, a simple probe trained over internal subject representations, succeeds at both tasks - correlating with both the QA accuracy of the model per-subject and FActScore, a recent factuality metric in open-ended generation.

Abstract

To evaluate knowledge in large language models (LLMs), current methods query the model and then evaluate its generated responses. In this work, we ask whether evaluation can be done before the model has generated any text. Concretely, is it possible to estimate how knowledgeable a model is about a certain entity, only from its internal computation? We study this question with two tasks: given a subject entity, the goal is to predict (a) the ability of the model to answer common questions about the entity, and (b) the factuality of open-ended responses generated by the model about the entity. Experiments with a variety of LLMs show that KEEN, a simple probe trained over internal subject representations, succeeds at both tasks - correlating with both the QA accuracy of the model per-subject and FActScore, a recent factuality metric in open-ended generation. Moreover, KEEN naturally aligns with the model's hedging behavior and faithfully reflects changes in the model's knowledge after fine-tuning. Lastly, we show a more interpretable yet equally performant variant of KEEN, which highlights a small set of tokens indicative of clusters and gaps in the model's knowledge. Being simple and lightweight, KEEN can be leveraged to guide decisions such as when it is appropriate to apply further training or augment queries with retrieval.

Figures (12)

• Figure 1: We show that simple probes (KEEN), trained over hidden model representations, quantify the model's knowledge about a given subject entity --- estimating the model's question-answering accuracy on entity-related questions (bottom left) and forecasting the factuality of model-generated texts about the entity (right).
• Figure 2: KEEN QA scores as a function of the fraction of per-subject queries that Vicuna 13B and Pythia 12B hedge on.
• Figure 3: Changes in the KEEN QA score and average QA accuracy after fine-tuning LLaMA2 7B on paragraphs about a target subject. These results are aggregated over individual fine-tuning processes for 20 target subjects.
• Figure 4: Predicted scores from the KEEN QA VP probe and the golden QA Accuracy scores are positively linearly related.
• Figure 5: Predicted scores of the KEEN OEG VP probe versus FActScore scores. KEEN scores are positively linearly correlated with FActScore scores.