Updating Parametric Knowledge with Context Distillation Retains Post-Training Capabilities

TL;DR

DiSC presents a principled method for continual knowledge updating of post-trained LLMs by recasting adaptation as context distillation. A frozen teacher, derived from the post-trained model, and a trainable student are fed document prefixes and suffixes split from update texts; the student minimizes the KL divergence between the teacher's prefix-conditioned suffix distribution and its own unconditioned suffix distribution. Across four post-trained models and two update domains, DiSC achieves stronger domain adaptation with substantially less forgetting of post-training capabilities than standard finetuning and prior context-distillation baselines. The results demonstrate that KL-based context distillation using split contexts is a practical and robust approach to maintaining post-training capabilities while incorporating new knowledge into large language models.

Abstract

Post-training endows pretrained LLMs with a variety of desirable skills, including instruction-following, reasoning, and others. However, these post-trained LLMs only encode knowledge up to a cut-off date, necessitating continual adaptation. Unfortunately, existing solutions cannot simultaneously learn new knowledge from an adaptation document corpora and mitigate the forgetting of earlier learned capabilities. To address this, we introduce Distillation via Split Contexts (DiSC), a simple context-distillation based approach for continual knowledge adaptation. \methodname~derives student and teacher distributions by conditioning on distinct segments of the training example and minimizes the KL divergence between the shared tokens. This allows us to efficiently apply context-distillation without requiring explicit generation steps during training. We run experiments on four post-trained models and two adaptation domains. Compared to prior finetuning and distillation methods for continual adaptation, DiSC consistently reports the best trade-off between learning new knowledge and mitigating forgetting of previously learned skills like instruction-following, reasoning, and factual knowledge.

Figures (2)

• Figure 1: Distillation via Split Contexts. First, we sample multiple split points and extract corresponding prefixes and suffixes. Then, we minimize the KL over the suffix distributions with context conditioning (teacher model $M_T$) and without (student model $M_S$). Our training only updates the student model $M_S$.
• Figure 2: IFEval vs KUP performances at different learning rates on Qwen-2.5-7B- Instruct for standard finetuning and DiSC.