Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CLoCKDistill: Consistent Location-and-Context-aware Knowledge Distillation for DETRs

Qizhen Lan, Qing Tian

TL;DR

CLoCKDistill addresses the challenge of compressing DETR detectors by combining memory-based feature distillation of the encoder with target-aware logit distillation guided by ground-truth objects. It leverages DETR's global context, as opposed to distilling backbone CNN features, and uses GT information to direct distillation focus. The method demonstrates significant mAP gains across KITTI and COCO for DINO, DAB-DETR, and Deformable-DETR, with improvements ranging from 2.2% to 6.4% in mAP over strong baselines. This approach highlights the importance of transformer-specific context in KD for DETRs and provides a practical route to efficient DETR deployment in resource-constrained environments.

Abstract

Object detection has advanced significantly with Detection Transformers (DETRs). However, these models are computationally demanding, posing challenges for deployment in resource-constrained environments (e.g., self-driving cars). Knowledge distillation (KD) is an effective compression method widely applied to CNN detectors, but its application to DETR models has been limited. Most KD methods for DETRs fail to distill transformer-specific global context. Also, they blindly believe in the teacher model, which can sometimes be misleading. To bridge the gaps, this paper proposes Consistent Location-and-Context-aware Knowledge Distillation (CLoCKDistill) for DETR detectors, which includes both feature distillation and logit distillation components. For feature distillation, instead of distilling backbone features like existing KD methods, we distill the transformer encoder output (i.e., memory) that contains valuable global context and long-range dependencies. Also, we enrich this memory with object location details during feature distillation so that the student model can prioritize relevant regions while effectively capturing the global context. To facilitate logit distillation, we create target-aware queries based on the ground truth, allowing both the student and teacher decoders to attend to consistent and accurate parts of encoder memory. Experiments on the KITTI and COCO datasets show our CLoCKDistill method's efficacy across various DETRs, e.g., single-scale DAB-DETR, multi-scale deformable DETR, and denoising-based DINO. Our method boosts student detector performance by 2.2% to 6.4%.

CLoCKDistill: Consistent Location-and-Context-aware Knowledge Distillation for DETRs

TL;DR

CLoCKDistill addresses the challenge of compressing DETR detectors by combining memory-based feature distillation of the encoder with target-aware logit distillation guided by ground-truth objects. It leverages DETR's global context, as opposed to distilling backbone CNN features, and uses GT information to direct distillation focus. The method demonstrates significant mAP gains across KITTI and COCO for DINO, DAB-DETR, and Deformable-DETR, with improvements ranging from 2.2% to 6.4% in mAP over strong baselines. This approach highlights the importance of transformer-specific context in KD for DETRs and provides a practical route to efficient DETR deployment in resource-constrained environments.

Abstract

Object detection has advanced significantly with Detection Transformers (DETRs). However, these models are computationally demanding, posing challenges for deployment in resource-constrained environments (e.g., self-driving cars). Knowledge distillation (KD) is an effective compression method widely applied to CNN detectors, but its application to DETR models has been limited. Most KD methods for DETRs fail to distill transformer-specific global context. Also, they blindly believe in the teacher model, which can sometimes be misleading. To bridge the gaps, this paper proposes Consistent Location-and-Context-aware Knowledge Distillation (CLoCKDistill) for DETR detectors, which includes both feature distillation and logit distillation components. For feature distillation, instead of distilling backbone features like existing KD methods, we distill the transformer encoder output (i.e., memory) that contains valuable global context and long-range dependencies. Also, we enrich this memory with object location details during feature distillation so that the student model can prioritize relevant regions while effectively capturing the global context. To facilitate logit distillation, we create target-aware queries based on the ground truth, allowing both the student and teacher decoders to attend to consistent and accurate parts of encoder memory. Experiments on the KITTI and COCO datasets show our CLoCKDistill method's efficacy across various DETRs, e.g., single-scale DAB-DETR, multi-scale deformable DETR, and denoising-based DINO. Our method boosts student detector performance by 2.2% to 6.4%.

Paper Structure

This paper contains 18 sections, 11 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Issues in feature-level distillation
  3. Issues in distilling DETR logits
  4. Related Work
  5. Visual Object Detection
  6. Knowledge Distillation for Object Detectors
  7. Methodology
  8. Location-and-context-aware Memory Distillation
  9. Target-aware Consistent Logit Distillation
  10. Experiments and Results
  11. Datasets
  12. Implementation Details
  13. Quantitative Results
  14. Qualitative Results
  15. Ablation Study
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Active regions of CNN-based and DETR-based detectors: (a) Image w/ bounding boxes, (b) ATSS zhang2020bridging and (c) Deformable DETR zhu2020deformable.
  • Figure 2: Overview of the proposed Consistent Location-and-Context-aware Knowledge Distillation for DETRs.
  • Figure 3: Attention maps of (a) an example COCO image from the (b) teacher, (c) student baseline, (d) KD-DETR distilled wang2024knowledge, and (e) our distilled models. Our distilled model allocates more focus to relevant objects and exhibits more clearly defined boundaries. Red indicates the highest attention and blue the lowest.