Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Integrating Diverse Assignment Strategies into DETRs

Yiwei Zhang, Jin Gao, Hanshi Wang, Fudong Ge, Guan Luo, Weiming Hu, Zhipeng Zhang

TL;DR

This work tackles the slow convergence of DETR-style detectors caused by strict one-to-one matching by revealing that diversity in one-to-many assignment strategies, not their quantity, drives performance gains. It introduces LoRA-DETR, a lightweight framework that attaches multiple LoRA-based auxiliary branches to the decoder FFN during training, each implementing a different one-to-many supervision rule, while discarding these branches at inference to keep cost unchanged. Through a unified design, LoRA-DETR demonstrates robust gains across baselines (e.g., Deformable-DETR and Relation-DETR) and achieves state-of-the-art or competitive results with improved training efficiency, such as reaching 52.1 AP after 24 epochs on COCO in some settings. The approach highlights a new paradigm for enriching supervision in DETR-like detectors by leveraging diverse, parameter-efficient auxiliary tasks without architectural complexity at inference.

Abstract

Label assignment is a critical component in object detectors, particularly within DETR-style frameworks where the one-to-one matching strategy, despite its end-to-end elegance, suffers from slow convergence due to sparse supervision. While recent works have explored one-to-many assignments to enrich supervisory signals, they often introduce complex, architecture-specific modifications and typically focus on a single auxiliary strategy, lacking a unified and scalable design. In this paper, we first systematically investigate the effects of ``one-to-many'' supervision and reveal a surprising insight that performance gains are driven not by the sheer quantity of supervision, but by the diversity of the assignment strategies employed. This finding suggests that a more elegant, parameter-efficient approach is attainable. Building on this insight, we propose LoRA-DETR, a flexible and lightweight framework that seamlessly integrates diverse assignment strategies into any DETR-style detector. Our method augments the primary network with multiple Low-Rank Adaptation (LoRA) branches during training, each instantiating a different one-to-many assignment rule. These branches act as auxiliary modules that inject rich, varied supervisory gradients into the main model and are discarded during inference, thus incurring no additional computational cost. This design promotes robust joint optimization while maintaining the architectural simplicity of the original detector. Extensive experiments on different baselines validate the effectiveness of our approach. Our work presents a new paradigm for enhancing detectors, demonstrating that diverse ``one-to-many'' supervision can be integrated to achieve state-of-the-art results without compromising model elegance.

Integrating Diverse Assignment Strategies into DETRs

TL;DR

This work tackles the slow convergence of DETR-style detectors caused by strict one-to-one matching by revealing that diversity in one-to-many assignment strategies, not their quantity, drives performance gains. It introduces LoRA-DETR, a lightweight framework that attaches multiple LoRA-based auxiliary branches to the decoder FFN during training, each implementing a different one-to-many supervision rule, while discarding these branches at inference to keep cost unchanged. Through a unified design, LoRA-DETR demonstrates robust gains across baselines (e.g., Deformable-DETR and Relation-DETR) and achieves state-of-the-art or competitive results with improved training efficiency, such as reaching 52.1 AP after 24 epochs on COCO in some settings. The approach highlights a new paradigm for enriching supervision in DETR-like detectors by leveraging diverse, parameter-efficient auxiliary tasks without architectural complexity at inference.

Abstract

Label assignment is a critical component in object detectors, particularly within DETR-style frameworks where the one-to-one matching strategy, despite its end-to-end elegance, suffers from slow convergence due to sparse supervision. While recent works have explored one-to-many assignments to enrich supervisory signals, they often introduce complex, architecture-specific modifications and typically focus on a single auxiliary strategy, lacking a unified and scalable design. In this paper, we first systematically investigate the effects of ``one-to-many'' supervision and reveal a surprising insight that performance gains are driven not by the sheer quantity of supervision, but by the diversity of the assignment strategies employed. This finding suggests that a more elegant, parameter-efficient approach is attainable. Building on this insight, we propose LoRA-DETR, a flexible and lightweight framework that seamlessly integrates diverse assignment strategies into any DETR-style detector. Our method augments the primary network with multiple Low-Rank Adaptation (LoRA) branches during training, each instantiating a different one-to-many assignment rule. These branches act as auxiliary modules that inject rich, varied supervisory gradients into the main model and are discarded during inference, thus incurring no additional computational cost. This design promotes robust joint optimization while maintaining the architectural simplicity of the original detector. Extensive experiments on different baselines validate the effectiveness of our approach. Our work presents a new paradigm for enhancing detectors, demonstrating that diverse ``one-to-many'' supervision can be integrated to achieve state-of-the-art results without compromising model elegance.
Paper Structure (12 sections, 6 equations, 3 figures, 7 tables)

This paper contains 12 sections, 6 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminary
  5. LoRA Branches in Decoder Layer
  6. Assignment Strategies
  7. Assignment Quality Loss
  8. Experiments
  9. Implementation Details
  10. Main Results
  11. Ablation Study
  12. Conclusion

Figures (3)

  • Figure 1: Different Decoder Architectures. (a) Standard DETR decoder layer, supervised solely by the one-to-one Hungarian matching strategy. (b) MS-DETR design that exchanges the positions of the cross-attention and self-attention modules, enabling supervision from both one-to-one and one-to-many assignment strategies, where cross-attention is guided by the latter. (c) A naive multi-branch extension of (a), where multiple FFNs are independently supervised by different assignment strategies. (d) Our proposed LoRA-DETR, which integrates lightweight LoRA branches to support multiple assignment strategies with minimal structural overhead. For clarity, the figure simplifies the merge between LoRA and FFN; see \ref{['eq:lora-ffn-merge']} for details. In all designs, only the primary branch output is fed to the next decoder layer.
  • Figure 2: A set of queries simultaneously participates in diverse matching strategies. (a) illustrates the primary branch using one-to-one Hungarian matching, while (b), (c), and (d) depict auxiliary branches employing one-to-many matching with different hyperparameter of $k$.
  • Figure 3: One-to-one training loss of the primary branch. (a) Varying the number of auxiliary branches. (b) Comparing different architectural variants, as defined in \ref{['fig:detr-arch-cmp']}. The x-axis denotes epochs and the y-axis denotes the loss value.