What Matters in Memorizing and Recalling Facts? Multifaceted Benchmarks for Knowledge Probing in Language Models

TL;DR

A knowledge probing benchmark, BELIEF(ICL), is introduced to evaluate the knowledge recall ability of both encoder- and decoder-based pre-trained language models (PLMs) from diverse perspectives and validate the effectiveness of BELIEFs in comprehensively evaluating PLM's knowledge recall ability on diverse PLMs, including recent large language models (LLMs).

Abstract

Language models often struggle with handling factual knowledge, exhibiting factual hallucination issue. This makes it vital to evaluate the models' ability to recall its parametric knowledge about facts. In this study, we introduce a knowledge probing benchmark, BELIEF(ICL), to evaluate the knowledge recall ability of both encoder- and decoder-based pre-trained language models (PLMs) from diverse perspectives. BELIEFs utilize a multi-prompt dataset to evaluate PLM's accuracy, consistency, and reliability in factual knowledge recall. To enable a more reliable evaluation with BELIEFs, we semi-automatically create MyriadLAMA, which has massively diverse prompts. We validate the effectiveness of BELIEFs in comprehensively evaluating PLM's knowledge recall ability on diverse PLMs, including recent large language models (LLMs). We then investigate key factors in memorizing and recalling facts in PLMs, such as model size, pretraining strategy and corpora, instruction-tuning process and in-context learning settings. Finally, we reveal the limitation of the prompt-based knowledge probing. The MyriadLAMA is publicized.

Figures (8)

• Figure 1: BELIEFs with MyriadLAMA: the BELIEF benchmarks utilize diverse factual prompts (here, MyriadLAMA) to assess LM's knowledge recall ability in terms of accuracy, consistency, and reliability.
• Figure 2: Calibration between confidence and Acc@1. (Left: Three BERT models types; Right: Llama3-8B with four different ICL settings.)
• Figure 3: Calibration plots of Llama2-7B (left) and Llama2-7B-IT (right).)
• Figure 4: The knowledge sharing rate between models and ICL settings. Each cell indicates the rate of facts correctly predicted by the top-listed model relative to those captured by the left-listed model.
• Figure 5: The relationship between the factual knowledge that can be covered by relation and template.