TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models

Abstract

To embed domain-specific or specialized knowledge into pre-trained foundation models, fine-tuning using techniques such as parameter efficient fine-tuning (e.g. LoRA) is a common practice. However, as new LLM architectures and pre-trained models emerge, transferring this specialized knowledge to newer models becomes an important task. In many scenarios, the original specialized data may be unavailable due to privacy or commercial restrictions, necessitating distillation and transfer of this specialized knowledge from the fine-tuned base model to a different pre-trained model. We present TuneShift-KD, a novel approach that automatically distills specialized knowledge from a fine-tuned model to a target model using only a few examples representative of the specialized information. Our key insight is that specialized knowledge can be identified through perplexity differences between base and fine-tuned models: prompts where the fine-tuned model responds confidently (low perplexity), but the base model struggles (high perplexity), indicate queries corresponding to the specialized knowledge learned by the fine-tuned model. TuneShift-KD leverages this insight to create a synthetic training dataset to transfer the specialized knowledge. Using an iterative process, TuneShift-KD generates more prompts similar to those that generated responses with specialized knowledge. TuneShift-KD does not require training discriminators or access to training datasets. It is an automated approach that only requires the initial fine-tuned and base models and a few representative prompts. Our experiments demonstrate that models fine-tuned using TuneShift-KD achieve higher accuracy than prior approaches, enabling ease of deployment and more effective transfer of the specialized knowledge.

Figures (10)

• Figure 1: TuneShift-KD Framework for Specialized Knowledge Transfer. This figure illustrates our approach to transferring specialized knowledge without requiring full access to original fine-tuning data. (a) The Synthetic Fine-tuning Dataset begins with just 5 seed examples from the original fine-tuning data and expands through our iterative process. (b) An instruction-tuned LLM generates similar prompts based on the prompt pattern "Generate 20 more samples like these 5 [the 5 sample data]". (c) Each synthetic prompt is fed to both the Source Base LLM and Source Fine-tuned LLM to obtain paired responses $\mathbf{y}^{(b)}$ and $\mathbf{y}^{(f)}$. (d) Our key filtering mechanism selects prompts where the fine-tuned model shows high confidence (low perplexity $\mathrm{PPL}(\mathbf{y}^{(f)})$) while the base model shows low confidence (high perplexity $\mathrm{PPL}(\mathbf{y}^{(b)})$), indicating the prompt targets specialized knowledge. Qualified prompt-response pairs are added to the synthetic dataset, while others are discarded. (e) The curated synthetic dataset is used to perform knowledge distillation, transferring specialized capabilities to the Target LLM without requiring original training data.
• Figure 2: t-SNE visualization of MBPP prompt embeddings generated by Llama2-13B versus GPT-4o. Embeddings computed using the MPNet encoder model demonstrate the superior diversity of GPT-4o generated prompts.
• Figure 3: Perplexity distribution of GSM8K examples generated by fine-tuned (Teacher) and base Llama2-7B models. Perplexities beyond the rightmost bin are omitted for clarity.
• Figure 4: Perplexity distribution of MBPP examples generated by fine-tuned (Teacher) and base Llama2-7B models. Perplexities beyond the rightmost bin are omitted for clarity.
• Figure 5: GSM8K example generated by fine-tuned and base Llama2-7B, with 1.19 fine-tuned perplexity and 4.22 base perplexity. The fine-tuned response is technically correct, and the base response is incorrect. The example was kept through the perplexity filter process.