Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Models

TL;DR

The paper tackles the knowledge gaps of static, high-parameter LLMs by introducing Knowledge Card, a modular framework of domain-specific, collaboratively trained knowledge cards that generate background content for a base LLM. Three content selectors enforce relevance, brevity, and factuality, and two integration strategies—Bottom-Up and Top-Down—enable flexible, multi-domain knowledge synthesis and proactive knowledge querying. Empirical results across MMLU, misinformation detection, and MidtermQA demonstrate substantial improvements over vanilla LLMs, generation-based prompting, and retrieval-augmented baselines, including strong temporal knowledge updates with far fewer parameters than retraining. The approach supports community-driven knowledge expansion and offers a path toward dynamic, updatable, domain-aware LLMs with reduced computational burden and carbon footprint.

Abstract

By design, large language models (LLMs) are static general-purpose models, expensive to retrain or update frequently. As they are increasingly adopted for knowledge-intensive tasks, it becomes evident that these design choices lead to failures to generate factual, relevant, and up-to-date knowledge. To this end, we propose Knowledge Card, a modular framework to plug in new factual and relevant knowledge into general-purpose LLMs. We first introduce knowledge cards -- specialized language models trained on corpora from specific domains and sources. Knowledge cards serve as parametric repositories that are selected at inference time to generate background knowledge for the base LLM. We then propose three content selectors to dynamically select and retain information in documents generated by knowledge cards, specifically controlling for relevance, brevity, and factuality of outputs. Finally, we propose two complementary integration approaches to augment the base LLM with the (relevant, factual) knowledge curated from the specialized LMs. Through extensive experiments, we demonstrate that Knowledge Card achieves state-of-the-art performance on six benchmark datasets. Ultimately, Knowledge Card framework enables dynamic synthesis and updates of knowledge from diverse domains. Its modularity will ensure that relevant knowledge can be continuously updated through the collective efforts of the research community.

Figures (9)

• Figure 1: Overview of Knowledge Card. We train knowledge cards on various knowledge domains and employ three knowledge selectors for quality control. We propose bottom-up and top-down to integrate general-purpose LLMs with modular and specialized LMs for multi-domain knowledge synthesis (bottom-up) and proactively seeking external knowledge (top-down).
• Figure 2: Performance on misinformation detection when each knowledge card is separately added. Knowledge Card enables modular patching of LLMs while in-domain knowledge cards help the most.
• Figure 3: Ablation study of the three knowledge selectors on misinformation detection. While the three selectors all contribute to model performance, the factuality selector is most crucial.
• Figure 4: Investigating the impact of $n_1$, $n_2$, and $n_3$, which govern the knowledge stream from modular knowledge cards to general-purpose LLMs. These hyperparameters enable fine-grained control over the knowledge synthesis process.
• Figure 5: Knowledge Card is compatible with other LLMs, specifically text-davinci-003 and gpt-3.5-turbo.