Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity

TL;DR

This work tackles the cost and inefficiency of multi-teacher knowledge distillation by introducing Angular-KD, which generates multiple semantically diverse views from a single teacher via lightweight view augmentation heads. It introduces two angular diversity objectives—Constrained Inter-angle Diversity Loss and Intra-angle Diversity Loss—to create diverse yet aligned augmented perspectives and prove that increased ensemble diversity tightens the upper bound on the ensemble loss, improving the student. The approach is validated across a broad suite of benchmarks (including CIFAR-100 and ImageNet) and shows strong gains over state-of-the-art augmentation methods while remaining plug-and-play with existing KD frameworks. The results indicate that angularly diverse, single-teacher augmentation provides a scalable and generalizable path to more effective KD with reduced computational overhead, with implications for deployment on resource-constrained devices.

Abstract

Knowledge Distillation (KD) aims to train a lightweight student model by transferring knowledge from a large, high-capacity teacher. Recent studies have shown that leveraging diverse teacher perspectives can significantly improve distillation performance; however, achieving such diversity typically requires multiple teacher networks, leading to high computational costs. In this work, we propose a novel cost-efficient knowledge augmentation method for KD that generates diverse multi-views by attaching multiple branches to a single teacher. To ensure meaningful semantic variation across multi-views, we introduce two angular diversity objectives: 1) constrained inter-angle diversify loss, which maximizes angles between augmented views while preserving proximity to the original teacher output, and 2) intra-angle diversify loss, which encourages an even distribution of views around the original output. The ensembled knowledge from these angularly diverse views, along with the original teacher, is distilled into the student. We further theoretically demonstrate that our objectives increase the diversity among ensemble members and thereby reduce the upper bound of the ensemble's expected loss, leading to more effective distillation. Experimental results show that our method surpasses an existing knowledge augmentation method across diverse configurations. Moreover, the proposed method is compatible with other KD frameworks in a plug-and-play fashion, providing consistent improvements in generalization performance.

Figures (4)

• Figure 1: The illustration of our angularly diverse knowledge augmentation: (a) Multi-views are generated from a single pre-trained teacher using multiple pathways, and (b) these augmented outputs are then optimized with two complementary angular objectives to maximize their intra- and inter-angular diversity.
• Figure 2: Few‐Shot Results.
• Figure 3:
• Figure 4: Correlation Matrix between augmented views ($V_1, \dots, V_5$) compared to TeKAP.