Self-Distillation Enables Continual Learning

TL;DR

Self-Distillation Fine-Tuning (SDFT) addresses continual learning for foundation models by enabling on-policy learning from demonstrations without explicit reward signals. It achieves this by using a demonstration-conditioned version of the same model as a teacher and distilling to a student via a reverse KL objective, yielding on-policy updates that preserve prior capabilities while acquiring new skills. Across skill learning and knowledge acquisition, SDFT outperforms supervised fine-tuning, delivering higher new-task accuracy and substantially reduced forgetting, and enabling sequential accumulation of multiple skills within a single model. The work demonstrates that on-policy distillation from demonstrations is a practical and effective path for continual learning, with potential to harmonize with reward-based RL and to scale with model capacity.

Abstract

Continual learning, enabling models to acquire new skills and knowledge without degrading existing capabilities, remains a fundamental challenge for foundation models. While on-policy reinforcement learning can reduce forgetting, it requires explicit reward functions that are often unavailable. Learning from expert demonstrations, the primary alternative, is dominated by supervised fine-tuning (SFT), which is inherently off-policy. We introduce Self-Distillation Fine-Tuning (SDFT), a simple method that enables on-policy learning directly from demonstrations. SDFT leverages in-context learning by using a demonstration-conditioned model as its own teacher, generating on-policy training signals that preserve prior capabilities while acquiring new skills. Across skill learning and knowledge acquisition tasks, SDFT consistently outperforms SFT, achieving higher new-task accuracy while substantially reducing catastrophic forgetting. In sequential learning experiments, SDFT enables a single model to accumulate multiple skills over time without performance regression, establishing on-policy distillation as a practical path to continual learning from demonstrations.

Figures (8)

• Figure 1: Supervised Fine-Tuning (SFT) is commonly used to learn from expert demonstration datasets, but its off-policy nature leads to catastrophic forgetting of general capabilities. We introduce Self-Distillation Fine-Tuning (SDFT), which turns expert demonstrations into on-policy learning signals by using a demonstration-conditioned version of the model as its own teacher. In this way, SDFT enables true continual learning with the model improving on new tasks as they arise without regressing existing capabilities.
• Figure 2: (Left) SDFT leverages a model’s in-context learning ability to generate on-policy training signals. For each query $x$, the model acts in two roles. A student that is conditioned only on the query $P=\pi(\cdot|x)$ and the teacher, which is the same model conditioned on an expert demonstration $c$, producing a demonstration-aware distribution $Q=\pi(\cdot|x,c)$. Training minimizes the reverse KL divergence between the student and teacher, yielding on-policy updates. (Right) Conditioning the model on the expert demonstrations creates a teacher with an output distribution that is substantially closer to the base model, while maintaining the same new-task accuracy.
• Figure 3: In a challenging continual learning experiment, where one model is trained sequentially on three different tasks, SDFT is able to learn each one while retaining performance on the others. In contrast, SFT performance on each task drops once it starts learning the next one. Performance is linearly normalized such that 0 corresponds to the base model accuracy on each one of the tasks, and 1 to the maximum accuracy obtained across both algorithms.
• Figure 4: Performance trade-offs between new task accuracy and retention of prior capabilities. Each point represents a trained model, with the top-right indicating ideal performance (high accuracy on both new and previous tasks). SDFT consistently achieves superior Pareto efficiency compared to baselines across all three skill learning tasks.
• Figure 5: (Left) SDFT benefits from model scale. Performance gap between SDFT and SFT on the Science Q&A task increases with model size, as larger models have stronger in-context learning capabilities. (Right) SDFT improves pass@k across various k, indicating genuine skill acquisition rather than entropy collapse.