Knowledgeable In-Context Tuning: Exploring and Exploiting Factual Knowledge for In-Context Learning

TL;DR

This work tackles the challenge that factual knowledge substantially shapes in-context learning (ICL) performance. It introduces Knowledgeable In-Context Tuning (KICT), a modular framework that (i) injects factual knowledge during pre-training via three self-supervised tasks (MEP, EDG, KQA), (ii) selects in-context demonstrations using a knowledge-aware retrieval mechanism, and (iii) calibrates predictions with knowledge-base priors. Across eight text classification and four QA tasks, KICT yields significant gains over strong baselines, with notable improvements when integrating all three components. The findings highlight three knowledge facets—inherent model knowledge, knowledge embedded in prompts, and knowledge biases in generation—that together determine ICL effectiveness, offering a practical, plug-and-play path to boost few-shot learning in autoregressive LLMs. The approach promises broader impact for real-world NLP tasks by making ICL more data-efficient and knowledge-aware.

Abstract

Large language models (LLMs) enable in-context learning (ICL) by conditioning on a few labeled training examples as a text-based prompt, eliminating the need for parameter updates and achieving competitive performance. In this paper, we demonstrate that factual knowledge is imperative for the performance of ICL in three core facets: the inherent knowledge learned in LLMs, the factual knowledge derived from the selected in-context examples, and the knowledge biases in LLMs for output generation. To unleash the power of LLMs in few-shot learning scenarios, we introduce a novel Knowledgeable In-Context Tuning (KICT) framework to further improve the performance of ICL: 1) injecting knowledge into LLMs during continual self-supervised pre-training, 2) judiciously selecting the examples for ICL with high knowledge relevance, and 3) calibrating the prediction results based on prior knowledge. We evaluate the proposed approaches on autoregressive models (e.g., GPT-style LLMs) over multiple text classification and question-answering tasks. Experimental results demonstrate that KICT substantially outperforms strong baselines and improves by more than 13% and 7% on text classification and question-answering tasks, respectively.

Figures (11)

• Figure 1: An example of in-context learning (ICL).
• Figure 2: Results of different scales of GPT-2 and OPT models over 8 text classification tasks and 4 question answering tasks in various component destruction settings. For each target example, we have $K=8$ labeled samples as the demonstration. Results indicate that factual knowledge is crucial to the performance of ICL.
• Figure 3: 4-shot results of GPT-2 (urge) over AGNews and TREC. For each frequency region, we sample top-5 label words for each category and report the accuracy for all label mapping permutations.
• Figure 4: The overview of the KICT framework. We introduce multiple plug-and-play knowledgeable techniques to enhance the utilization of knowledge for improving ICL performance. Left: We propose three knowledge-aware self-supervised learning tasks that infuse factual knowledge into LLMs during pre-training. Middle: We utilize entity-related information to select in-context examples that exhibit high knowledge relevance to the target example. Right: For prediction, we derive prior information from large-scale corpora to calibrate the predictions.
• Figure 5: GPT-2 (large) sample effectiveness (%) of (only w. KER) with different values of $K$.