Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Powerful Teachers Matter: Text-Guided Multi-view Knowledge Distillation with Visual Prior Enhancement

Xin Zhang, Jianyang Xu, Hao Peng, Dongjing Wang, Jingyuan Zheng, Yu Li, Yuyu Yin, Hongbo Wang

Abstract

Knowledge distillation transfers knowledge from large teacher models to smaller students for efficient inference. While existing methods primarily focus on distillation strategies, they often overlook the importance of enhancing teacher knowledge quality. In this paper, we propose Text-guided Multi-view Knowledge Distillation (TMKD), which leverages dual-modality teachers, a visual teacher and a text teacher (CLIP), to provide richer supervisory signals. Specifically, we enhance the visual teacher with multi-view inputs incorporating visual priors (edge and high-frequency features), while the text teacher generates semantic weights through prior-aware prompts to guide adaptive feature fusion. Additionally, we introduce vision-language contrastive regularization to strengthen semantic knowledge in the student model. Extensive experiments on five benchmarks demonstrate that TMKD consistently improves knowledge distillation performance by up to 4.49\%, validating the effectiveness of our dual-teacher multi-view enhancement strategy. Code is available at https://anonymous.4open.science/r/TMKD-main-44D1.

Powerful Teachers Matter: Text-Guided Multi-view Knowledge Distillation with Visual Prior Enhancement

Abstract

Knowledge distillation transfers knowledge from large teacher models to smaller students for efficient inference. While existing methods primarily focus on distillation strategies, they often overlook the importance of enhancing teacher knowledge quality. In this paper, we propose Text-guided Multi-view Knowledge Distillation (TMKD), which leverages dual-modality teachers, a visual teacher and a text teacher (CLIP), to provide richer supervisory signals. Specifically, we enhance the visual teacher with multi-view inputs incorporating visual priors (edge and high-frequency features), while the text teacher generates semantic weights through prior-aware prompts to guide adaptive feature fusion. Additionally, we introduce vision-language contrastive regularization to strengthen semantic knowledge in the student model. Extensive experiments on five benchmarks demonstrate that TMKD consistently improves knowledge distillation performance by up to 4.49\%, validating the effectiveness of our dual-teacher multi-view enhancement strategy. Code is available at https://anonymous.4open.science/r/TMKD-main-44D1.
Paper Structure (26 sections, 12 equations, 6 figures, 6 tables)

This paper contains 26 sections, 12 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Knowledge Distillation
  4. VLMs for Knowledge Transfer
  5. Multi-Teacher Distillation
  6. Method
  7. Multi-view Images Construction
  8. Edge Enhanced View Generation
  9. High-Frequency Enhanced View Generation
  10. Text-guided Feature Fusion
  11. View-specific Prompts Construction
  12. Semantic-Guided Adaptive Fusion
  13. Distillation Loss Function
  14. Feature-level Distillation
  15. Logits-level Distillation
  16. ...and 11 more sections

Figures (6)

  • Figure 1: (a) Single-view Distillation: Attention incorrectly focuses on the stone rather than the bird. (b) Our Multi-view Distillation: Attention correctly targets the bird, demonstrating that the model captures task-relevant features.
  • Figure 2: Overview of our TMKD framework. TMKD employs dual-modality teachers: (1) a visual teacher that extracts features from multi-view inputs and fuses them with semantic weights through prior-aware prompts, and (2) a text teacher (CLIP). The visual teacher transfers knowledge through feature-level and logits-level distillation, while the text teacher provides semantic guidance via vision-language contrastive representation distillation (CRD) loss, enabling effective transfer of both visual and semantic knowledge to the student model.
  • Figure 3: Attention heatmap visualization. (a)(d) Input image. (b)(e) Attention map of our proposed method. (c)(f) Attention map of the baseline method CAT-KD.
  • Figure 4: Scatter plots of Student, CAT-KD, and CAT-KD+ours on CUB-200. Our method combined with CAT-KD generates tighter and more separated clusters, indicating improved class-wise consistency and feature discriminability.
  • Figure 5: Class-wise logit heatmaps and distribution.
  • ...and 1 more figures