Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs

TL;DR

The paper tackles the problem of knowledge conflicts and high resource demands in multi-teacher knowledge distillation for LLMs. It introduces Knowledge Purification to condense diverse teacher rationales into a single, cohesive rationale and proposes five purification methods spanning aggregation, routing, and RL-based selection. Through extensive experiments on commonsense and biomedical reasoning, the authors show that purification improves distillation performance and mitigates conflicts, with router-based approaches offering robust out-of-domain generalization. The findings support a practical path toward efficient, powerful lightweight models via multi-teacher distillation guided by purified knowledge.

Abstract

Knowledge distillation has emerged as a pivotal technique for transferring knowledge from stronger large language models (LLMs) to smaller, more efficient models. However, traditional distillation approaches face challenges related to knowledge conflicts and high resource demands, particularly when leveraging multiple teacher models. In this paper, we introduce the concept of \textbf{Knowledge Purification}, which consolidates the rationales from multiple teacher LLMs into a single rationale, thereby mitigating conflicts and enhancing efficiency. To investigate the effectiveness of knowledge purification, we further propose five purification methods from various perspectives. Our experiments demonstrate that these methods not only improve the performance of the distilled model but also effectively alleviate knowledge conflicts. Moreover, router-based methods exhibit robust generalization capabilities, underscoring the potential of innovative purification techniques in optimizing multi-teacher distillation and facilitating the practical deployment of powerful yet lightweight models.

Figures (5)

• Figure 1: Effects of increasing teacher LLMs on the performance of the TinyLLM framework.
• Figure 2: An illustration of five knowledge purification methods proposed in our work.
• Figure 3: Evaluation of knowledge purification methods with an increasing number of teacher LLMs. We visualize the CMV of knowledge aggregation as an example, represented by the signed area of the shaded region (positive when above TinyLLM and negative when below).
• Figure 4: Prompt used for GPT-4 to perform knowledge aggregation, consisted of global instruction, in-context example, and query.
• Figure 5: An example on the OBQA dataset. Five proposed methods are used for knowledge purification.

Theorems & Definitions (2)

• Proposition 1
• proof