Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mean Teacher DETR with Masked Feature Alignment: A Robust Domain Adaptive Detection Transformer Framework

Weixi Weng, Chun Yuan

TL;DR

MTM tackles unsupervised domain adaptation for Detection Transformer (DETR) by introducing a two-stage framework that uses labeled target-like images for robust pretraining and mean-teacher self-training on unlabeled target images. It couples Object Queries Knowledge Transfer (OQKT) with masked feature alignment (MDQFA, MTWFA) to stabilize training and reduce domain shift, achieving consistent gains across weather, scene, and synthetic-to-real adaptations. The approach is supported by ablation studies and a theoretical bound on target error, demonstrating both practical improvements and sound reasoning for the design. Overall, MTM offers a robust, domain-robust DETR-based UDAOD framework with a principled use of a third related domain to enhance adaptation.

Abstract

Unsupervised domain adaptation object detection (UDAOD) research on Detection Transformer(DETR) mainly focuses on feature alignment and existing methods can be divided into two kinds, each of which has its unresolved issues. One-stage feature alignment methods can easily lead to performance fluctuation and training stagnation. Two-stage feature alignment method based on mean teacher comprises a pretraining stage followed by a self-training stage, each facing problems in obtaining reliable pretrained model and achieving consistent performance gains. Methods mentioned above have not yet explore how to utilize the third related domain such as target-like domain to assist adaptation. To address these issues, we propose a two-stage framework named MTM, i.e. Mean Teacher-DETR with Masked Feature Alignment. In the pretraining stage, we utilize labeled target-like images produced by image style transfer to avoid performance fluctuation. In the self-training stage, we leverage unlabeled target images by pseudo labels based on mean teacher and propose a module called Object Queries Knowledge Transfer (OQKT) to ensure consistent performance gains of the student model. Most importantly, we propose masked feature alignment methods including Masked Domain Query-based Feature Alignment (MDQFA) and Masked Token-wise Feature Alignment (MTWFA) to alleviate domain shift in a more robust way, which not only prevent training stagnation and lead to a robust pretrained model in the pretraining stage, but also enhance the model's target performance in the self-training stage. Experiments on three challenging scenarios and a theoretical analysis verify the effectiveness of MTM.

Mean Teacher DETR with Masked Feature Alignment: A Robust Domain Adaptive Detection Transformer Framework

TL;DR

MTM tackles unsupervised domain adaptation for Detection Transformer (DETR) by introducing a two-stage framework that uses labeled target-like images for robust pretraining and mean-teacher self-training on unlabeled target images. It couples Object Queries Knowledge Transfer (OQKT) with masked feature alignment (MDQFA, MTWFA) to stabilize training and reduce domain shift, achieving consistent gains across weather, scene, and synthetic-to-real adaptations. The approach is supported by ablation studies and a theoretical bound on target error, demonstrating both practical improvements and sound reasoning for the design. Overall, MTM offers a robust, domain-robust DETR-based UDAOD framework with a principled use of a third related domain to enhance adaptation.

Abstract

Unsupervised domain adaptation object detection (UDAOD) research on Detection Transformer(DETR) mainly focuses on feature alignment and existing methods can be divided into two kinds, each of which has its unresolved issues. One-stage feature alignment methods can easily lead to performance fluctuation and training stagnation. Two-stage feature alignment method based on mean teacher comprises a pretraining stage followed by a self-training stage, each facing problems in obtaining reliable pretrained model and achieving consistent performance gains. Methods mentioned above have not yet explore how to utilize the third related domain such as target-like domain to assist adaptation. To address these issues, we propose a two-stage framework named MTM, i.e. Mean Teacher-DETR with Masked Feature Alignment. In the pretraining stage, we utilize labeled target-like images produced by image style transfer to avoid performance fluctuation. In the self-training stage, we leverage unlabeled target images by pseudo labels based on mean teacher and propose a module called Object Queries Knowledge Transfer (OQKT) to ensure consistent performance gains of the student model. Most importantly, we propose masked feature alignment methods including Masked Domain Query-based Feature Alignment (MDQFA) and Masked Token-wise Feature Alignment (MTWFA) to alleviate domain shift in a more robust way, which not only prevent training stagnation and lead to a robust pretrained model in the pretraining stage, but also enhance the model's target performance in the self-training stage. Experiments on three challenging scenarios and a theoretical analysis verify the effectiveness of MTM.
Paper Structure (24 sections, 1 theorem, 17 equations, 2 figures, 4 tables)

This paper contains 24 sections, 1 theorem, 17 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Detection Transformer
  4. Domain Adaptive Object Detection
  5. Methods
  6. MTM Framework Overview
  7. Object Queries Knowledge Transfer
  8. Masked Feature Alignment
  9. Masked Domain Query-based Feature Alignment
  10. Masked Token-Wise Feature Alignment
  11. Overall Training Strategy
  12. Experiment
  13. Datasets and Settings
  14. Datasets
  15. Comparative Benchmarks
  16. ...and 9 more sections

Key Result

Theorem 1

For any $\delta \in (0,1)$, with probability $1-\delta$,

Figures (2)

  • Figure 1: We train Deformable DETR from scratch with different pretraining strategies in the weather adaptation scenario. It turns out that pretraining with masked feature alignment on source images and target-like images brings the best performance.
  • Figure 2: MTM framework in the self-training stage. In the mean teacher structure, the student model is updated by back-propagation while the teacher model is updated by the Exponential Moving Average (EMA) of the student model. OQKT enhances the student's object queries with the teacher's object queries by the multi-head attention mechanism. Masked feature alignment is also conducted in this stage to alleviate the domain shift between source data and target data.

Theorems & Definitions (1)

  • Theorem 1