Towards Reliable Latent Knowledge Estimation in LLMs: Zero-Prompt Many-Shot Based Factual Knowledge Extraction

TL;DR

This work tackles the challenge of reliably estimating factual knowledge latent in LLMs by eliminating prompt engineering. It introduces Zero-Prompt Latent Knowledge Estimator (ZP-LKE), which uses many-shot in-context examples of (subject, object) pairs to elicit the model's knowledge without explicit prompts. Across 49 open-source LLMs and 50 relations from Wikidata/T-REx, ZP-LKE outperforms traditional prompt-based probing, revealing robust cross-model knowledge structures and illustrating the effects of model size and instruction fine-tuning on latent knowledge. The study provides a scalable, model-agnostic framework for comparing latent knowledge across LLMs and offers practical guidance on in-context example design, with careful attention to ethical considerations.

Abstract

In this paper, we focus on the challenging task of reliably estimating factual knowledge that is embedded inside large language models (LLMs). To avoid reliability concerns with prior approaches, we propose to eliminate prompt engineering when probing LLMs for factual knowledge. Our approach, called Zero-Prompt Latent Knowledge Estimator (ZP-LKE), leverages the in-context learning ability of LLMs to communicate both the factual knowledge question as well as the expected answer format. Our knowledge estimator is both conceptually simpler (i.e., doesn't depend on meta-linguistic judgments of LLMs) and easier to apply (i.e., is not LLM-specific), and we demonstrate that it can surface more of the latent knowledge embedded in LLMs. We also investigate how different design choices affect the performance of ZP-LKE. Using the proposed estimator, we perform a large-scale evaluation of the factual knowledge of a variety of open-source LLMs, like OPT, Pythia, Llama(2), Mistral, Gemma, etc. over a large set of relations and facts from the Wikidata knowledge base. We observe differences in the factual knowledge between different model families and models of different sizes, that some relations are consistently better known than others but that models differ in the precise facts they know, and differences in the knowledge of base models and their finetuned counterparts. Code available at: https://github.com/QinyuanWu0710/ZeroPrompt_LKE

Figures (16)

• Figure 1: Overview of how Latent Knowledge Estimators (LKEs) work
• Figure 2: Current prompt-based (zero-shot and few-shot) LKE approaches vs. Our zero-prompt (many-shot) LKE approach
• Figure 3: Impact of in-context example count on multiple-choice accuracy across LLMs. The dashed line marks the number needed for 95% stable accuracy with 50 examples.
• Figure 4: Variation in Nobel laureate data probabilities using Mistral-7B. Figure \ref{['fig:correct_sequence']} illustrates object probabilities at various positions in the prompt. Figures \ref{['fig:unknown_distributed']} and \ref{['fig:unknown_simultaneous']} show impacts of unknown objects at random and continuous positions, while Figures \ref{['fig:incorrect_distributed']} and \ref{['fig:incorrect_simultaneous']} show effects of incorrect examples. The dashed line indicates average correct probabilities (blue dots).
• Figure 5: Comparison of LKEs using response and multiple-choice accuracy across 12 relations from T-REx-MC. ZP-LKE is evaluated against the baseline method jiang2020can.

Theorems & Definitions (1)

• Example 1