Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DINO-X: A Unified Vision Model for Open-World Object Detection and Understanding

Tianhe Ren, Yihao Chen, Qing Jiang, Zhaoyang Zeng, Yuda Xiong, Wenlong Liu, Zhengyu Ma, Junyi Shen, Yuan Gao, Xiaoke Jiang, Xingyu Chen, Zhuheng Song, Yuhong Zhang, Hongjie Huang, Han Gao, Shilong Liu, Hao Zhang, Feng Li, Kent Yu, Lei Zhang

TL;DR

DINO-X introduces a unified, open-world object-centric vision framework that extends Grounding DINO 1.5 with expanded prompts (text, visual, customized) and a large-scale grounding corpus (Grounding-100M) to support multi-task outputs including detection, segmentation, keypoints, and language-based understanding. The Pro model achieves state-of-the-art zero-shot performance on COCO and LVIS, with notable gains on rare long-tailed classes, while the Edge variant emphasizes real-time inference via EfficientViT and knowledge distillation. A two-stage training strategy and prompt-tuning enable robust open-world grounding and prompt-free detection, demonstrating strong cross-task capabilities such as region captioning and object-based QA. Overall, DINO-X advances open-world perception by delivering high accuracy across detection, segmentation, pose estimation, and language tasks, with practical applicability on edge devices.

Abstract

In this paper, we introduce DINO-X, which is a unified object-centric vision model developed by IDEA Research with the best open-world object detection performance to date. DINO-X employs the same Transformer-based encoder-decoder architecture as Grounding DINO 1.5 to pursue an object-level representation for open-world object understanding. To make long-tailed object detection easy, DINO-X extends its input options to support text prompt, visual prompt, and customized prompt. With such flexible prompt options, we develop a universal object prompt to support prompt-free open-world detection, making it possible to detect anything in an image without requiring users to provide any prompt. To enhance the model's core grounding capability, we have constructed a large-scale dataset with over 100 million high-quality grounding samples, referred to as Grounding-100M, for advancing the model's open-vocabulary detection performance. Pre-training on such a large-scale grounding dataset leads to a foundational object-level representation, which enables DINO-X to integrate multiple perception heads to simultaneously support multiple object perception and understanding tasks, including detection, segmentation, pose estimation, object captioning, object-based QA, etc. Experimental results demonstrate the superior performance of DINO-X. Specifically, the DINO-X Pro model achieves 56.0 AP, 59.8 AP, and 52.4 AP on the COCO, LVIS-minival, and LVIS-val zero-shot object detection benchmarks, respectively. Notably, it scores 63.3 AP and 56.5 AP on the rare classes of LVIS-minival and LVIS-val benchmarks, improving the previous SOTA performance by 5.8 AP and 5.0 AP. Such a result underscores its significantly improved capacity for recognizing long-tailed objects.

DINO-X: A Unified Vision Model for Open-World Object Detection and Understanding

TL;DR

DINO-X introduces a unified, open-world object-centric vision framework that extends Grounding DINO 1.5 with expanded prompts (text, visual, customized) and a large-scale grounding corpus (Grounding-100M) to support multi-task outputs including detection, segmentation, keypoints, and language-based understanding. The Pro model achieves state-of-the-art zero-shot performance on COCO and LVIS, with notable gains on rare long-tailed classes, while the Edge variant emphasizes real-time inference via EfficientViT and knowledge distillation. A two-stage training strategy and prompt-tuning enable robust open-world grounding and prompt-free detection, demonstrating strong cross-task capabilities such as region captioning and object-based QA. Overall, DINO-X advances open-world perception by delivering high accuracy across detection, segmentation, pose estimation, and language tasks, with practical applicability on edge devices.

Abstract

In this paper, we introduce DINO-X, which is a unified object-centric vision model developed by IDEA Research with the best open-world object detection performance to date. DINO-X employs the same Transformer-based encoder-decoder architecture as Grounding DINO 1.5 to pursue an object-level representation for open-world object understanding. To make long-tailed object detection easy, DINO-X extends its input options to support text prompt, visual prompt, and customized prompt. With such flexible prompt options, we develop a universal object prompt to support prompt-free open-world detection, making it possible to detect anything in an image without requiring users to provide any prompt. To enhance the model's core grounding capability, we have constructed a large-scale dataset with over 100 million high-quality grounding samples, referred to as Grounding-100M, for advancing the model's open-vocabulary detection performance. Pre-training on such a large-scale grounding dataset leads to a foundational object-level representation, which enables DINO-X to integrate multiple perception heads to simultaneously support multiple object perception and understanding tasks, including detection, segmentation, pose estimation, object captioning, object-based QA, etc. Experimental results demonstrate the superior performance of DINO-X. Specifically, the DINO-X Pro model achieves 56.0 AP, 59.8 AP, and 52.4 AP on the COCO, LVIS-minival, and LVIS-val zero-shot object detection benchmarks, respectively. Notably, it scores 63.3 AP and 56.5 AP on the rare classes of LVIS-minival and LVIS-val benchmarks, improving the previous SOTA performance by 5.8 AP and 5.0 AP. Such a result underscores its significantly improved capacity for recognizing long-tailed objects.

Paper Structure

This paper contains 41 sections, 11 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. Model Architecture
  4. DINO-X Pro
  5. Text Prompt Encoder:
  6. Visual Prompt Encoder:
  7. Customized Prompt:
  8. Box Head:
  9. Mask Head:
  10. Keypoint Head:
  11. Language Head:
  12. DINO-X Edge
  13. Stronger Text Prompt Encoder:
  14. Knowledge Distillation:
  15. Improved FP16 Inference:
  16. ...and 26 more sections

Figures (11)

  • Figure 1: DINO-X is a unified object-centric vision model which supports various open-world perception and object-level understanding tasks, including Open-World Object Detection and Segmentation, Phrase Grounding, Visual Prompt Counting, Pose Estimation, Prompt-Free Object Detection and Recognition, Dense Region Caption, etc.
  • Figure 2: DINO-X Pro zero-shot performance on public detection benchmarks. Comparing with Grounding DINO 1.5 Pro and Grounding DINO 1.6 Pro, DINO-X Pro achieves new state-of-the-art (SOTA) performance on COCO, LVIS-minival, and LVIS-val zero-shot benchmarks. Furthermore, it outperforms other models with larger margins in detecting rare classes of objects on LVIS-minival and LVIS-val, demonstrating its exceptional capability of recognizing long-tailed objects.
  • Figure 3: DINO-X is designed to accept text prompt, visual prompt, and customized prompt, and is capable of simultaneously generating outputs ranging from coarse-level representations, such as bounding boxes, to fine-grained details, including masks, keypoints, and object captions.
  • Figure 4: The detailed design of language head in DINO-X. It involves using a frozen DINO-X to extract object tokens, and a linear projection aligns its dimensions with the text embeddings. The lightweight language decoder then integrates these object and task tokens to generate response outputs in an autoregressive manner. The task tokens equip the language decoder with the capability of tackling different tasks.
  • Figure 5: Open-world object detection with DINO-X
  • ...and 6 more figures