SiNGER: A Clearer Voice Distills Vision Transformers Further

TL;DR

SiNGER tackles the problem of high-norm artifacts in Vision Transformer (ViT) representations that bias knowledge distillation toward uninformative tokens. It introduces a singular nullspace-guided energy reallocation framework, implemented via a lightweight LoRA-based adapter, to refine teacher features by perturbations constrained to the left-nullspace of the next transformer block, thereby suppressing artifacts while preserving informative signals. The method uses three losses—$\mathcal{L}_{KD}$, $\mathcal{L}_{outlier}$, and $\mathcal{L}_{info}$—across selected distillation layers to achieve artifact-aware transfer and more interpretable representations. Across six downstream tasks, SiNGER delivers consistent improvements over prior KD methods and yields representations that resemble the teacher's structure without copying artifacts, enabling closer-to-teacher performance in smaller students and enabling more robust cross-task transfer.

Abstract

Vision Transformers are widely adopted as the backbone of vision foundation models, but they are known to produce high-norm artifacts that degrade representation quality. When knowledge distillation transfers these features to students, high-norm artifacts dominate the objective, so students overfit to artifacts and underweight informative signals, diminishing the gains from larger models. Prior work attempted to remove artifacts but encountered an inherent trade-off between artifact suppression and preserving informative signals from teachers. To address this, we introduce Singular Nullspace-Guided Energy Reallocation (SiNGER), a novel distillation framework that suppresses artifacts while preserving informative signals. The key idea is principled teacher feature refinement: during refinement, we leverage the nullspace-guided perturbation to preserve information while suppressing artifacts. Then, the refined teacher's features are distilled to a student. We implement this perturbation efficiently with a LoRA-based adapter that requires minimal structural modification. Extensive experiments show that \oursname consistently improves student models, achieving state-of-the-art performance in multiple downstream tasks and producing clearer and more interpretable representations.

Figures (11)

• Figure 1: SiNGER suppresses artifacts and enhances transfer. (a) Feature visualizations highlight clearer and more interpretable representations. (b) Radar chart shows consistent multi-task gains.
• Figure 2: Qualitative analysis. Row 1: KD method comparison. Left: distilled feature map colored by patch norm, Right: patch-wise cosine similarity to the teacher. Row 2: Input image, two pretrained ViTs, and three ViT-L ${\to}$ ViT-T distilled variants. Each panel shows similarity from the $\times$-marked patch. SiNGER most closely preserves teacher semantics, showing the most coherent teacher-consistent similarity patterns.
• Figure 3: The overall pipeline of knowledge distillation with the SiNGER adapter at $l$th layer .
• Figure 4: Two objectives of SiNGER; (a) outlier suppression and (b) information preservation. $\Vert F \Vert$ in (a) is signed with the cosine-similarity between $F$ and $F$. In (b), the cosine similarity between $\times$-marked patch and every patch of another feature map is visualized.
• Figure 5: Sub-layer change analysis at two depths. Top: box-plots of relative changes/gains in each layer. Bottom: summary statistics in each layer.