Lightweight Model Pre-training via Language Guided Knowledge Distillation
Mingsheng Li, Lin Zhang, Mingzhen Zhu, Zilong Huang, Gang Yu, Jiayuan Fan, Tao Chen
TL;DR
This work introduces Language-Guided Distillation (LGD) to pre-train lightweight computer vision models without labels by leveraging task-oriented language guidance. LGD builds a Textual Semantics Bank (TSB) from a pre-trained text encoder and a momentum-updated Visual Semantics Bank (VSB) via Language-Guided Knowledge Aggregation (LGKA), then applies two cross-space alignment losses to drive the student to imitate the teacher in both visual and textual spaces. The resulting framework, with a simple joint objective $L_{LGD} = \alpha L_{VIS} + (1-\alpha) L_{TEX}$, demonstrates state-of-the-art transfer to zero-shot, semi-/fully-supervised classification, object detection, instance segmentation, and semantic segmentation across multiple datasets and backbone pairs. The approach also supports task-specific text control and generalizes to various image encoders, highlighting its practical potential for efficient, mobile-friendly model pre-training.
Abstract
This paper studies the problem of pre-training for small models, which is essential for many mobile devices. Current state-of-the-art methods on this problem transfer the representational knowledge of a large network (as a Teacher) into a smaller model (as a Student) using self-supervised distillation, improving the performance of the small model on downstream tasks. However, existing approaches are insufficient in extracting the crucial knowledge that is useful for discerning categories in downstream tasks during the distillation process. In this paper, for the first time, we introduce language guidance to the distillation process and propose a new method named Language-Guided Distillation (LGD) system, which uses category names of the target downstream task to help refine the knowledge transferred between the teacher and student. To this end, we utilize a pre-trained text encoder to extract semantic embeddings from language and construct a textual semantic space called Textual Semantics Bank (TSB). Furthermore, we design a Language-Guided Knowledge Aggregation (LGKA) module to construct the visual semantic space, also named Visual Semantics Bank (VSB). The task-related knowledge is transferred by driving a student encoder to mimic the similarity score distribution inferred by a teacher over TSB and VSB. Compared with other small models obtained by either ImageNet pre-training or self-supervised distillation, experiment results show that the distilled lightweight model using the proposed LGD method presents state-of-the-art performance and is validated on various downstream tasks, including classification, detection, and segmentation. We have made the code available at https://github.com/mZhenz/LGD.