Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Unsupervised Model Selection for Domain Adaptive Object Detection

Hengfu Yu, Jinhong Deng, Wen Li, Lixin Duan

TL;DR

This work tackles the problem of selecting near-optimal models for Domain Adaptive Object Detection (DAOD) without target-domain labels. It introduces Detection Adaptation Score (DAS), a principled unsupervised metric built on the flat-minima hypothesis, composed of a Flatness Index Score ($\mathrm{FIS}$) and a Prototypical Distance Ratio ($\mathrm{PDR}$). The authors ground DAS in a domain adaptation generalization bound that links flatness to model variance and domain-distance to minima, and they derive explicit formulations for $\mathrm{FIS}$ and $\mathrm{PDR}$ to quantify transferability and discriminability. Through extensive experiments on Real-to-Art, Weather, and Synthetic-to-Real DAOD benchmarks, DAS demonstrates strong correlation with target performance (e.g., $\text{PCC} \approx 0.86$) and outperforms existing unsupervised model-evaluation methods, enabling effective checkpoint selection without target labels.

Abstract

Evaluating the performance of deep models in new scenarios has drawn increasing attention in recent years. However, while it is possible to collect data from new scenarios, the annotations are not always available. Existing DAOD methods often rely on validation or test sets on the target domain for model selection, which is impractical in real-world applications. In this paper, we propose a novel unsupervised model selection approach for domain adaptive object detection, which is able to select almost the optimal model for the target domain without using any target labels. Our approach is based on the flat minima principle, i,e., models located in the flat minima region in the parameter space usually exhibit excellent generalization ability. However, traditional methods require labeled data to evaluate how well a model is located in the flat minima region, which is unrealistic for the DAOD task. Therefore, we design a Detection Adaptation Score (DAS) approach to approximately measure the flat minima without using target labels. We show via a generalization bound that the flatness can be deemed as model variance, while the minima depend on the domain distribution distance for the DAOD task. Accordingly, we propose a Flatness Index Score (FIS) to assess the flatness by measuring the classification and localization fluctuation before and after perturbations of model parameters and a Prototypical Distance Ratio (PDR) score to seek the minima by measuring the transferability and discriminability of the models. In this way, the proposed DAS approach can effectively evaluate the model generalization ability on the target domain. We have conducted extensive experiments on various DAOD benchmarks and approaches, and the experimental results show that the proposed DAS correlates well with the performance of DAOD models and can be used as an effective tool for model selection after training.

Towards Unsupervised Model Selection for Domain Adaptive Object Detection

TL;DR

This work tackles the problem of selecting near-optimal models for Domain Adaptive Object Detection (DAOD) without target-domain labels. It introduces Detection Adaptation Score (DAS), a principled unsupervised metric built on the flat-minima hypothesis, composed of a Flatness Index Score () and a Prototypical Distance Ratio (). The authors ground DAS in a domain adaptation generalization bound that links flatness to model variance and domain-distance to minima, and they derive explicit formulations for and to quantify transferability and discriminability. Through extensive experiments on Real-to-Art, Weather, and Synthetic-to-Real DAOD benchmarks, DAS demonstrates strong correlation with target performance (e.g., ) and outperforms existing unsupervised model-evaluation methods, enabling effective checkpoint selection without target labels.

Abstract

Evaluating the performance of deep models in new scenarios has drawn increasing attention in recent years. However, while it is possible to collect data from new scenarios, the annotations are not always available. Existing DAOD methods often rely on validation or test sets on the target domain for model selection, which is impractical in real-world applications. In this paper, we propose a novel unsupervised model selection approach for domain adaptive object detection, which is able to select almost the optimal model for the target domain without using any target labels. Our approach is based on the flat minima principle, i,e., models located in the flat minima region in the parameter space usually exhibit excellent generalization ability. However, traditional methods require labeled data to evaluate how well a model is located in the flat minima region, which is unrealistic for the DAOD task. Therefore, we design a Detection Adaptation Score (DAS) approach to approximately measure the flat minima without using target labels. We show via a generalization bound that the flatness can be deemed as model variance, while the minima depend on the domain distribution distance for the DAOD task. Accordingly, we propose a Flatness Index Score (FIS) to assess the flatness by measuring the classification and localization fluctuation before and after perturbations of model parameters and a Prototypical Distance Ratio (PDR) score to seek the minima by measuring the transferability and discriminability of the models. In this way, the proposed DAS approach can effectively evaluate the model generalization ability on the target domain. We have conducted extensive experiments on various DAOD benchmarks and approaches, and the experimental results show that the proposed DAS correlates well with the performance of DAOD models and can be used as an effective tool for model selection after training.

Paper Structure

This paper contains 18 sections, 1 theorem, 12 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Domain Adaptation Generalization Bound with Flat Minima
  5. Flatness Index Score
  6. Prototypical Distance Ratio
  7. Detection Adaptation Score
  8. Experiments
  9. Experiment Setup
  10. Main Results
  11. Further Analysis
  12. Conclusion
  13. Appendix
  14. Proofs of Theorem 1
  15. More Implementation Details
  16. ...and 3 more sections

Key Result

Theorem 1

Given any $\delta\ge 0$, exist hypothesis $h \in \mathcal{H}$ where $\cH$ is the hypothesis set, $\theta_{h}$ denotes the parameters of $h$. Given any hypothesis $h' \in \{h' | h'\in \mathcal{H}, \| \theta_{h'}-\theta_{h} \|_2 \le \tau \}$, which is located in the neighborhood of $h$ with radius $\t where $dis(\mathcal{S},\mathcal{T})$ is the distribution mismatch between the source domain $\mathc

Figures (7)

  • Figure 1: (a) The performance of classic DAOD method AT AT on Real-to-Art (P2C) adaptation task during training. It suffers from performance degradation as the training goes on. The proposed DAS outperforms previous unsupervised model evaluation methods and selects desirable checkpoints without accessing any labels in the target domain. (b) The motivation of the work. We propose a Detection Adaptation Score including a Prototypical Distance Ratio (PDR) score and Flatness Index Score (FIS) to evaluate the model performance in an unsupervised way. It can be a good substitute metric for using annotations for DAOD model evaluation.
  • Figure 2: The comparison of different unsupervised model evaluation methods for DAOD. The experiments are conducted on real-to-art adaptation (P2C) using AT. Note that the directions of all scores are unified.
  • Figure 3: Hyperprameter tuning on AT AT using our DAS. (a) $\lambda_{\mathrm{dis}}$ that controls the weight of adversarial loss from domain discriminator. (b)$\lambda_{\mathrm{unsup}}$ that controls the weight of unsupervised loss.
  • Figure 4: Ablation study of the proposed DAS method. The results are averaged from DAOD benchmarks and approaches.
  • Figure 5: The hyperparameter sensitivity of the proposed method on real-to-art adaptation.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Theorem 1
  • proof