FiGKD: Fine-Grained Knowledge Distillation via High-Frequency Detail Transfer

TL;DR

FiGKD introduces a frequency-aware knowledge distillation framework that applies a 2D Discrete Wavelet Transform to the teacher’s and student’s logit vectors, separating them into low-frequency content and high-frequency detail. By distilling only the high-frequency components, FiGKD captures subtle, fine-grained semantic distinctions essential for distinguishing visually similar classes, while relying on ground-truth labels for coarse class identity. Empirical results across CIFAR-100, TinyImageNet, and several FGVR benchmarks show FiGKD consistently surpasses both logit-based and feature-based distillation methods, with particularly large gains on fine-grained tasks. The approach is architecture-agnostic, requires no intermediate feature maps, and demonstrates robustness across homogeneous and heterogeneous teacher–student pairs, indicating strong practical relevance for resource-constrained deployment.

Abstract

Knowledge distillation (KD) is a widely adopted technique for transferring knowledge from a high-capacity teacher model to a smaller student model by aligning their output distributions. However, existing methods often underperform in fine-grained visual recognition tasks, where distinguishing subtle differences between visually similar classes is essential. This performance gap stems from the fact that conventional approaches treat the teacher's output logits as a single, undifferentiated signal-assuming all contained information is equally beneficial to the student. Consequently, student models may become overloaded with redundant signals and fail to capture the teacher's nuanced decision boundaries. To address this issue, we propose Fine-Grained Knowledge Distillation (FiGKD), a novel frequency-aware framework that decomposes a model's logits into low-frequency (content) and high-frequency (detail) components using the discrete wavelet transform (DWT). FiGKD selectively transfers only the high-frequency components, which encode the teacher's semantic decision patterns, while discarding redundant low-frequency content already conveyed through ground-truth supervision. Our approach is simple, architecture-agnostic, and requires no access to intermediate feature maps. Extensive experiments on CIFAR-100, TinyImageNet, and multiple fine-grained recognition benchmarks show that FiGKD consistently outperforms state-of-the-art logit-based and feature-based distillation methods across a variety of teacher-student configurations. These findings confirm that frequency-aware logit decomposition enables more efficient and effective knowledge transfer, particularly in resource-constrained settings.

Figures (6)

• Figure 1: Comparison of teacher, conventional KD, and FiGKD models on a fine-grained classification task. When the teacher accurately captures fine-grained distinctions, the student trained with conventional KD fails to replicate this behavior. In contrast, the FiGKD student succeeds by focusing on high-frequency detail signals.
• Figure 2: Original logits (Left), low-frequency logits (Middle), and high-frequency logits (Right) obtained from (Top) ResNet32x4 (79.42%) and (Bottom) ResNet8x4 (72.50%) in CIFAR-100. ResNet32x4 predicts the correct class from both the original and high-frequency logits, whereas ResNet8x4 fails to do so using the high-frequency logits alone, despite predicting correctly from the original logits.
• Figure 3: Overview of Fine-Grained Knowledge Distillation (FiGKD) framework. Given an input image from a fine-grained recognition dataset (e.g., different species of albatross), the teacher and student models produce their respective logit vectors. Both logits are reshaped and decomposed using a 2D DW, yielding high-frequency components $\mathcal{F}^\mathcal{T}_H$ and $\mathcal{F}^\mathcal{S}_H$ for the teacher and student, respectively. The detail loss $\mathcal{L}_\text{detail}$ is then computed between these high-frequency components, enabling the student to learn fine-grained semantic distinctions from the teacher’s logit space. This targeted knowledge transfer facilitates improved recognition performance in visually similar classes.
• Figure 4: Accuracy sensitivity according to loss weight on the CIFAR-100 (Top) and TinyImageNet (Bottom). The blue line shows the results when the CE loss weight is fixed to 1 and the high-frequency loss weight is varied. The orange line represents the results when the high-frequency loss weight is fixed to 2 and the CE loss weight is varied.
• Figure 5: Original (Left), low-frequency (Middle), and high-frequency (Right) logits for different model pairs on CIFAR-100. While large models correctly predict the target class using both the original and high-frequency logits, small models are only able to predict correctly with the original logits and fail to do so when relying solely on the high-frequency logits.