Knowledge Amalgamation for Object Detection with Transformers

TL;DR

A more effective KA scheme for Transformer-based object detection models is explored, considering the architecture characteristics of Transformers, and a hint is generated within the sequence-level amalgamation by concatenating teacher sequences instead of redundantly aggregating them to a fixed-size one as previous KA approaches.

Abstract

Knowledge amalgamation (KA) is a novel deep model reusing task aiming to transfer knowledge from several well-trained teachers to a multi-talented and compact student. Currently, most of these approaches are tailored for convolutional neural networks (CNNs). However, there is a tendency that transformers, with a completely different architecture, are starting to challenge the domination of CNNs in many computer vision tasks. Nevertheless, directly applying the previous KA methods to transformers leads to severe performance degradation. In this work, we explore a more effective KA scheme for transformer-based object detection models. Specifically, considering the architecture characteristics of transformers, we propose to dissolve the KA into two aspects: sequence-level amalgamation (SA) and task-level amalgamation (TA). In particular, a hint is generated within the sequence-level amalgamation by concatenating teacher sequences instead of redundantly aggregating them to a fixed-size one as previous KA works. Besides, the student learns heterogeneous detection tasks through soft targets with efficiency in the task-level amalgamation. Extensive experiments on PASCAL VOC and COCO have unfolded that the sequence-level amalgamation significantly boosts the performance of students, while the previous methods impair the students. Moreover, the transformer-based students excel in learning amalgamated knowledge, as they have mastered heterogeneous detection tasks rapidly and achieved superior or at least comparable performance to those of the teachers in their specializations.

Figures (8)

• Figure 1: The overall workflow of our proposed knowledge amalgamation method for transformer-based object detectors is shown in the two-teacher case. Our method aims to transfer knowledge from several well-trained teachers to one compact yet multi-talented student, which is explicitly capable of detecting all the categories taught by given teachers. As shown in the figure, our method is composed of two separate components. (1) Sequence-level amalgamation (SA): for the CNN backbone and all encoder layers, the intermediate sequence of the student is supervised by concatenated (and compressed) outputs of all corresponding teacher sequences. (2) Task-level amalgamation (TA): the student learns heterogenous detection tasks by mimicking the soft targets predicted by teachers. The gradient flows are shown as the dotted lines from our KA modules and the ground truth supervision.
• Figure 2: Illustration of sequence extension and sequence-level amalgamation for the student model. We duplicate the linear projection layer (after the CNN backbone) to generate the extended student sequences (e.g., $X_S = X_s^1 \oplus X_2^2$). In this way, the student sequences (e.g., $Y_S = Y_s^1 \oplus Y_s^2$) can be directly supervised by the corresponding teacher sequences (e.g., $Y_T = Y_t^1 \oplus Y_t^2$). Additionally, the sequence compression module is utilized to decrease the computation cost, as explained in Section \ref{['sec:seq-compression']}.
• Figure 3: Illustration of different sequence compression methods. The student sequences are supervised by partial teacher hints, compressed based on various sequence compression strategies, i.e., index set $P_\text{slim}$. However, we have kept the integrity of vision tokens (they are either preserved or discarded) to avoid introducing noise as the SAG approach.
• Figure 4: Average precision (COCO style) learning curves of our proposed method and baseline settings evaluated on PASCAL VOC and COCO datasets. Specifically, on VOC, we train all models for 150 epochs with the learning rate decayed at 100 epochs with relatively smaller image size than dai2020up-detr; on COCO, we train for only 60 epochs with learning rate decayed at 50 epochs. Besides, we plot AP learning curves of ablation study on loss terms and ground truth labels separately on VOC in (c). Here, SA denotes training with only sequence-level amalgamation term, and SAG denotes sequence aggregation method. TA represents training only with task-level amalgamation term, and LF means training without ground truth labels.
• Figure 5: Detection results on randomly sampled images from the validation dataset of COCO. The first two rows are detection results of two teachers respectively (with green, and red bounding boxes) and the final row contains the detection results of the student (with both green and red bounding boxes).

Theorems & Definitions (7)

• Definition 1: Token redundancy
• Lemma 1
• proof
• Theorem 1
• proof
• Theorem 2
• proof