Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations
Faisal Hamman, Pasan Dissanayake, Yanjun Fu, Sanghamitra Dutta
TL;DR
This work tackles the data scarcity challenge in task-aware KD for LLMs by introducing CoD, which augments few-shot training with counterfactual explanations to better capture the teacher’s decision boundary. It provides statistical and geometric guarantees to motivate the use of CFEs and describes a practical CFE-generation pipeline that leverages LLMs. Empirically, CoD improves distillation performance across multiple datasets and model families in few-shot settings, often halving the required real labeled data while maintaining or boosting accuracy. The approach offers a data-efficient, explainability-driven path to deploying task-focused, smaller LLMs in resource-constrained environments, with potential extensions to generative models.
Abstract
Knowledge distillation is a promising approach to transfer capabilities from complex teacher models to smaller, resource-efficient student models that can be deployed easily, particularly in task-aware scenarios. However, existing methods of task-aware distillation typically require substantial quantities of data which may be unavailable or expensive to obtain in many practical scenarios. In this paper, we address this challenge by introducing a novel strategy called Counterfactual-explanation-infused Distillation CoD for few-shot task-aware knowledge distillation by systematically infusing counterfactual explanations. Counterfactual explanations (CFEs) refer to inputs that can flip the output prediction of the teacher model with minimum perturbation. Our strategy CoD leverages these CFEs to precisely map the teacher's decision boundary with significantly fewer samples. We provide theoretical guarantees for motivating the role of CFEs in distillation, from both statistical and geometric perspectives. We mathematically show that CFEs can improve parameter estimation by providing more informative examples near the teacher's decision boundary. We also derive geometric insights on how CFEs effectively act as knowledge probes, helping the students mimic the teacher's decision boundaries more effectively than standard data. We perform experiments across various datasets and LLMs to show that CoD outperforms standard distillation approaches in few-shot regimes (as low as 8-512 samples). Notably, CoD only uses half of the original samples used by the baselines, paired with their corresponding CFEs and still improves performance.