Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection

Yuwei Zhang, Wenhao Yu, Shangbin Feng, Yifan Zhu, Letian Peng, Jayanth Srinivasa, Gaowen Liu, Jingbo Shang

TL;DR

This work introduces WikiDYK, a real-world, continuously evolving benchmark for evaluating knowledge injection into language models, derived from Wikipedia's expert-curated Did You Know entries. Through a multi-dimensional QA evaluation, the authors show that BiLMs consistently outperform CLMs in memorization tasks, with CLMs lagging by about 23 percentage points in reliability. They propose an ensemble framework that uses BiLMs as external knowledge repositories integrated with LLMs via a scoped routing mechanism, achieving up to 29.1% improvement in reliability. The results highlight the value of bidirectional architectures for knowledge-intensive tasks and offer a scalable, modular approach to maintain up-to-date knowledge without catastrophic forgetting.

Abstract

Despite significant advances in large language models (LLMs), their knowledge memorization capabilities remain underexplored, due to the lack of standardized and high-quality test ground. In this paper, we introduce a novel, real-world and large-scale knowledge injection benchmark that evolves continuously over time without requiring human intervention. Specifically, we propose WikiDYK, which leverages recently-added and human-written facts from Wikipedia's "Did You Know..." entries. These entries are carefully selected by expert Wikipedia editors based on criteria such as verifiability and clarity. Each entry is converted into multiple question-answer pairs spanning diverse task formats from easy cloze prompts to complex multi-hop questions. WikiDYK contains 12,290 facts and 77,180 questions, which is also seamlessly extensible with future updates from Wikipedia editors. Extensive experiments using continued pre-training reveal a surprising insight: despite their prevalence in modern LLMs, Causal Language Models (CLMs) demonstrate significantly weaker knowledge memorization capabilities compared to Bidirectional Language Models (BiLMs), exhibiting a 23% lower accuracy in terms of reliability. To compensate for the smaller scales of current BiLMs, we introduce a modular collaborative framework utilizing ensembles of BiLMs as external knowledge repositories to integrate with LLMs. Experiment shows that our framework further improves the reliability accuracy by up to 29.1%.

Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection

TL;DR

This work introduces WikiDYK, a real-world, continuously evolving benchmark for evaluating knowledge injection into language models, derived from Wikipedia's expert-curated Did You Know entries. Through a multi-dimensional QA evaluation, the authors show that BiLMs consistently outperform CLMs in memorization tasks, with CLMs lagging by about 23 percentage points in reliability. They propose an ensemble framework that uses BiLMs as external knowledge repositories integrated with LLMs via a scoped routing mechanism, achieving up to 29.1% improvement in reliability. The results highlight the value of bidirectional architectures for knowledge-intensive tasks and offer a scalable, modular approach to maintain up-to-date knowledge without catastrophic forgetting.

Abstract

Despite significant advances in large language models (LLMs), their knowledge memorization capabilities remain underexplored, due to the lack of standardized and high-quality test ground. In this paper, we introduce a novel, real-world and large-scale knowledge injection benchmark that evolves continuously over time without requiring human intervention. Specifically, we propose WikiDYK, which leverages recently-added and human-written facts from Wikipedia's "Did You Know..." entries. These entries are carefully selected by expert Wikipedia editors based on criteria such as verifiability and clarity. Each entry is converted into multiple question-answer pairs spanning diverse task formats from easy cloze prompts to complex multi-hop questions. WikiDYK contains 12,290 facts and 77,180 questions, which is also seamlessly extensible with future updates from Wikipedia editors. Extensive experiments using continued pre-training reveal a surprising insight: despite their prevalence in modern LLMs, Causal Language Models (CLMs) demonstrate significantly weaker knowledge memorization capabilities compared to Bidirectional Language Models (BiLMs), exhibiting a 23% lower accuracy in terms of reliability. To compensate for the smaller scales of current BiLMs, we introduce a modular collaborative framework utilizing ensembles of BiLMs as external knowledge repositories to integrate with LLMs. Experiment shows that our framework further improves the reliability accuracy by up to 29.1%.
Paper Structure (20 sections, 3 figures, 6 tables)

This paper contains 20 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. WikiDYK Benchmark
  4. Data Collection
  5. Evaluation
  6. Static Analysis
  7. Impact of Retrieval Augmented Generation (RAG)
  8. Knowledge Injection Approaches
  9. Knowledge Injection Preliminaries
  10. Continued Pretraining for CLMs
  11. Continued Pretraining for BiLMs
  12. Ensemble Pipeline
  13. Results and Analysis
  14. Experimental Setup
  15. Main Results
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Proposed knowledge injection evaluation workflow. We inject the knowledge from via continued pre-training which can be achieved via various model architectures or training objectives. The injected model is then evaluated with questions from multiple dimensions from easy cloze prompts to complex multi-hop questions. Notice that the images are not used in the dataset.
  • Figure 2: Effect of number of knowledge injected with number of upsampling $s=1000$.
  • Figure 3: Effect of number of upsampling with $1000$ injected knowledge.