Instruction-tuned Language Models are Better Knowledge Learners

TL;DR

The paper addresses the challenge of keeping LLM factual knowledge up-to-date by studying continual knowledge acquisition and identifying a perplexity-driven bottleneck. It proposes pre-instruction-tuning (PIT), which pre-trains on QA pairs before encoding information from documents, and demonstrates through extensive experiments that PIT outperforms standard instruction-tuning by a substantial margin (e.g., up to 17.8% on key metrics) and enhances cross-domain generalization. The work introduces Wiki2023 to reliably assess knowledge absorption from new documents, and conducts thorough ablations (including PIT++, domain transfer) to uncover that learning how to access knowledge via QA is more impactful than merely encoding dense documents. The findings offer a practical, scalable approach to improve up-to-date factual reasoning in LLMs, with implications for continual learning, domain adaptation, and alignment pipelines across real-world applications.

Abstract

In order for large language model (LLM)-based assistants to effectively adapt to evolving information needs, it must be possible to update their factual knowledge through continued training on new data. The standard recipe for doing so involves continued pre-training on new documents followed by instruction-tuning on question-answer (QA) pairs. However, we find that LLMs trained with this recipe struggle to answer questions, even though the perplexity of documents is minimized. We found that QA pairs are generally straightforward, while documents are more complex, weaving many factual statements together in an intricate manner. Therefore, we hypothesize that it is beneficial to expose LLMs to QA pairs before continued pre-training on documents so that the process of encoding knowledge from complex documents takes into account how this knowledge is accessed through questions. Based on this, we propose pre-instruction-tuning (PIT), a method that instruction-tunes on questions prior to training on documents. This contrasts with standard instruction-tuning, which learns how to extract knowledge after training on documents. Extensive experiments and ablation studies demonstrate that pre-instruction-tuning significantly enhances the ability of LLMs to absorb knowledge from new documents, outperforming standard instruction-tuning by 17.8%.

Figures (6)

• Figure 1: Illustration of continued pre-training (first row), continued pre-training followed by instruction-tuning (second row), and pre-instruction-tuning before continued pre-training (last row), along with their accuracies on evaluation questions. Each right-pointing light-blue triangle indicates a training phase.
• Figure 2: The Wiki2023 dataset. Top-right: the number of documents and QA pairs; Top-left: frequent keywords in questions; Bottom: the distribution of token counts in documents, questions, and answers.
• Figure 3: An example document about "Oppenheimer" and corresponding QA pairs from Wiki2023. Tokens used for computing losses are highlighted in green.
• Figure 4: Different experimental settings examined in this paper. Each row represents a different experimental setting with a unique name and number, and each vertical section highlighted by a right-pointing light-blue triangle indicates a training phase. Models are assessed on test QA across all settings. Whenever multiple datasets are enclosed within a dashed square, they are mixed together during the training process.
• Figure 5: We vary the number of epochs (\ref{['fig:epochs']}) and learning rate (\ref{['fig:lr']}) during continued pre-training to study the training dynamics of Llama-2 7B. The left axis is QA accuracies for test questions, measured by exact match. On the right axis, we display 2 metrics indicated by distinct colors: the perplexity of all tokens in the documents, and the knowledge retention accuracy, measured by QA accuracy on the Natural Questions dataset. We highlight situations where perplexity of all document tokens is minimized to 1.