CAE-DFKD: Bridging the Transferability Gap in Data-Free Knowledge Distillation

TL;DR

Data-free KD transfers knowledge without access to real data, but prior methods are limited by synthetic data quality and cross-domain transferability. CAE-DFKD introduces an embedding-level framework with a category-aware prior via language-model derived embeddings $\mathbf{E}^{\text{off}}$ and a Category Embedding Noise Diffusion (CEND) layer to enrich the embedding space, plus embedding-level Category Noise Contrastive Learning (CNCL) to promote domain-invariant features. Generator and student optimizations combine loss terms $\mathcal{L}_{G}=\mathcal{L}_{CE}+\lambda_{BN}\mathcal{L}_{BN}+\lambda_{adv}\mathcal{L}_{adv}$ and $\mathcal{L}_{S}=\mathcal{L}_{KL}+\alpha\mathcal{L}_{CNCL}$ to drive learning. Experiments on CIFAR-10/100, Tiny-ImageNet, ImageNet-1K, NYUv2, ADE-20K, and COCO-2017 show competitive recognition performance and improved downstream transfer, with CEND providing efficiency gains and CNCL improving generalization across tasks.

Abstract

Data-Free Knowledge Distillation (DFKD) enables the knowledge transfer from the given pre-trained teacher network to the target student model without access to the real training data. Existing DFKD methods focus primarily on improving image recognition performance on associated datasets, often neglecting the crucial aspect of the transferability of learned representations. In this paper, we propose Category-Aware Embedding Data-Free Knowledge Distillation (CAE-DFKD), which addresses at the embedding level the limitations of previous rely on image-level methods to improve model generalization but fail when directly applied to DFKD. The superiority and flexibility of CAE-DFKD are extensively evaluated, including: \textit{\textbf{i.)}} Significant efficiency advantages resulting from altering the generator training paradigm; \textit{\textbf{ii.)}} Competitive performance with existing DFKD state-of-the-art methods on image recognition tasks; \textit{\textbf{iii.)}} Remarkable transferability of data-free learned representations demonstrated in downstream tasks.

Figures (5)

• Figure 1: From KD to CAE-DFKD.$(a)$ The traditional paradigm of knowledge distillation (KD) has access to the real dataset. $(b)$ Data-Free Knowledge Distillation (DFKD) is unaware of the read dataset and employs synthetic datasets for knowledge transfer. $(c)$ The goal of CAE-DFKD is to continually transfer the knowledge acquired under data-free setting to downstream tasks.
• Figure 2: Quality Difference in Synthetic Images.$(a)$ The proportion of low-confidence ($\leq 0.1$ highest probability) synthetic images within the corresponding categories. $(b)$ The generator inevitably produces semantically ambiguous, low-quality synthetic images. $(c)$ Applying image-level noise interference or high-intensity data augmentation to uncertain images renders their semantics more abstract and difficult to exploit.
• Figure 3: CAE-DFKD Framework. (a) A pre-trained language model (LM, default CLIP) provides a category-structured initial embedding space, $\mathbf{E}^{\text{off}}$, contrasting with the un-structured Gaussian noise used in native DFKD, this process is performed offline and does not cause any training burden; (b) During generator updates, the Category Embedding Noise Diffusion (CEND) layer addresses $\mathbf{E}^{\text{off}}$ sparsity and lack of diversity. CEND samples from noise sources with distinct pre-set distributions, dynamically diffusing category embeddings to induce a diverse embedding space, $\mathbf{E}$. synthetic images, written to memory, are fed to teacher and student networks to compute cross-entropy ($\mathcal{L}_{ce}$), adversarial ($\mathcal{L}_{adv}$), and batch normalization ($\mathcal{L}_{bn}$) losses for generator updates; (c) During student updates, classic logit knowledge distillation ($\mathcal{L}_{kl}$) is performed on synthetic images read from memory. Further, Category Noise Contrastive Learning (CNCL) constructs embedding-level positive-negative pairs, optimizing towards $\mathcal{L}_{cncl}$.
• Figure 4: Category Embedding Noise Diffusion (CEND). We leverage a pre-trained language model to initialize category embeddings ($\mathbf{E}^{\text{off}}$) with structure, in contrast to the unstructured Gaussian noise used in native DFKD. However, $\mathbf{E}^{\text{off}}$ suffers from sparsity and lacks diversity. Our proposed CEND layer introduces $N$ noise sources ($\{NS\}$), each following a distinct pre-defined distribution (illustrated here with N=4), sampling from each $NS_n (n=1,\cdots4)$ and element-adding it to $\mathbf{E}^{\text{off}}$ enables dynamic diffusion of the initial embeddings, resulting in a richer embedding space $\mathbf{E}$.
• Figure 5: Downstream Task Performance Visualization Comparison. The contrastive learning designed for image-level optimization proves to be less effective in enhancing the generalization ability of the student network in DFKD (Section \ref{['Introduction']}). In contrast, our embedding-level contrastive learning encourages the student network to learn domain-invariant class features, which benefits both depth estimation and semantic segmentation.