Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LaMI-DETR: Open-Vocabulary Detection with Language Model Instruction

Penghui Du, Yu Wang, Yifan Sun, Luting Wang, Yue Liao, Gang Zhang, Errui Ding, Yan Wang, Jingdong Wang, Si Liu

TL;DR

LaMI-DETR tackles open-vocabulary object detection by addressing two core issues: limited concept representation in CLIP's text space and overfitting to base categories. It introduces Language Model Instruction (LaMI), which uses GPT to generate visual concepts and T5-based clustering to capture inter-category visual relationships, combined with Federated Loss-based visual concept sampling to encourage generalization. The LaMI-DETR detector integrates these language-derived cues into a DETR-like framework with a frozen CLIP backbone, achieving end-to-end training and a VLM-calibrated scoring scheme. Empirically, it delivers state-of-the-art results on OVOD benchmarks, notably $AP_r = 43.4$ on OV-LVIS—$7.8$ AP_r points above prior best—and demonstrates strong cross-dataset transfer, highlighting the practical impact of infusing structured language-derived knowledge into open-vocabulary detection.

Abstract

Existing methods enhance open-vocabulary object detection by leveraging the robust open-vocabulary recognition capabilities of Vision-Language Models (VLMs), such as CLIP.However, two main challenges emerge:(1) A deficiency in concept representation, where the category names in CLIP's text space lack textual and visual knowledge.(2) An overfitting tendency towards base categories, with the open vocabulary knowledge biased towards base categories during the transfer from VLMs to detectors.To address these challenges, we propose the Language Model Instruction (LaMI) strategy, which leverages the relationships between visual concepts and applies them within a simple yet effective DETR-like detector, termed LaMI-DETR.LaMI utilizes GPT to construct visual concepts and employs T5 to investigate visual similarities across categories.These inter-category relationships refine concept representation and avoid overfitting to base categories.Comprehensive experiments validate our approach's superior performance over existing methods in the same rigorous setting without reliance on external training resources.LaMI-DETR achieves a rare box AP of 43.4 on OV-LVIS, surpassing the previous best by 7.8 rare box AP.

LaMI-DETR: Open-Vocabulary Detection with Language Model Instruction

TL;DR

LaMI-DETR tackles open-vocabulary object detection by addressing two core issues: limited concept representation in CLIP's text space and overfitting to base categories. It introduces Language Model Instruction (LaMI), which uses GPT to generate visual concepts and T5-based clustering to capture inter-category visual relationships, combined with Federated Loss-based visual concept sampling to encourage generalization. The LaMI-DETR detector integrates these language-derived cues into a DETR-like framework with a frozen CLIP backbone, achieving end-to-end training and a VLM-calibrated scoring scheme. Empirically, it delivers state-of-the-art results on OVOD benchmarks, notably on OV-LVIS— AP_r points above prior best—and demonstrates strong cross-dataset transfer, highlighting the practical impact of infusing structured language-derived knowledge into open-vocabulary detection.

Abstract

Existing methods enhance open-vocabulary object detection by leveraging the robust open-vocabulary recognition capabilities of Vision-Language Models (VLMs), such as CLIP.However, two main challenges emerge:(1) A deficiency in concept representation, where the category names in CLIP's text space lack textual and visual knowledge.(2) An overfitting tendency towards base categories, with the open vocabulary knowledge biased towards base categories during the transfer from VLMs to detectors.To address these challenges, we propose the Language Model Instruction (LaMI) strategy, which leverages the relationships between visual concepts and applies them within a simple yet effective DETR-like detector, termed LaMI-DETR.LaMI utilizes GPT to construct visual concepts and employs T5 to investigate visual similarities across categories.These inter-category relationships refine concept representation and avoid overfitting to base categories.Comprehensive experiments validate our approach's superior performance over existing methods in the same rigorous setting without reliance on external training resources.LaMI-DETR achieves a rare box AP of 43.4 on OV-LVIS, surpassing the previous best by 7.8 rare box AP.
Paper Structure (35 sections, 7 equations, 6 figures, 8 tables)

This paper contains 35 sections, 7 equations, 6 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Open-vocabulary object detection (OVOD)
  4. Zero-shot object detection (ZSD)
  5. Large Language Model (LLM)
  6. Method
  7. Preliminaries
  8. Architecture of LaMI-DETR
  9. Comparison with other Open-Vocabulary DETR.
  10. Language Model Instruction
  11. Inter-category Relationships Extraction.
  12. Language Embedding Fusion.
  13. Confusing Category.
  14. Visual Concept Sampling.
  15. Comparison with concept enrichment.
  16. ...and 20 more sections

Figures (6)

  • Figure 1: Illustration of the concept representation challenge. The clustering results are from (a) name embeddings by CLIP text encoder, (b) name embeddings by T5, and (c) visual description embeddings by T5, respectively. (a) CLIP text encoder struggles to distinguish between category names that are compositionally similar in letters, such as "fireboat" and "fireweed". (b) T5 fails to cluster categories that are visually comparable but compositionally different in name around the same cluster center, such as "sea-lion" and "dugong". (c) Marrying T5's textual semantic knowledge with visual insights achieves reasonable cluster results.
  • Figure 2: An overview of LaMI-DETR Framework. LaMI-DETR adapts the DETR model by incorporating a frozen CLIP image encoder as the backbone and replacing the final classification layer with CLIP text embeddings. (a) Visual Concept Sampling, applied only during the training phase, leverages pre-extracted inter-category relationships to sample easy negative categories that are visually distinct from ground truth classes. This encourages the detector to derive more generalized foreground features rather than overfitting to base categories. (b) Language embeddings selected are integrated into the object queries for enhanced classification accuracy. (c) During inference, confusing categories are identified to improve VLM score.
  • Figure 3: Illustration of Inter-category Relationships Extraction. Visual descriptions generated by GPT-3.5 are processed by T5 to cluster categories with visual similarities.
  • Figure 4: Visualization of results by LaMI-DETR on OV-LVIS. For better clarity, we only display the prediction results for novel categories.
  • Figure 5: Visualization of proposals generated by the model with and without LaMI. Sequentially from top to bottom, each row displays the results for the ground-truth, LaMI-DETR, and the baseline, respectively. For detailed examination, please zoom in.
  • ...and 1 more figures