InstructDET: Diversifying Referring Object Detection with Generalized Instructions

TL;DR

This work tackles the practical gap in referring object detection (ROD) where existing REC data fails to cover diverse user intents. It introduces InstructDET, a data-centric framework that auto-generates generalized, human-like detection instructions using foundation models, yielding the InDET dataset built from existing REC and detection data. A conventional ROD model trained on InDET (DROD) demonstrates improved generalization and surpasses state-of-the-art visual grounding methods on InDET and standard benchmarks, while ablations show the value of global/local prompting and CLIP-based filtering. The approach significantly broadens the scope of detectable expressions, enabling robust handling of single- and multi-object references and offering practical implications for real-world object detection tasks and neural-symbolic reasoning systems.

Abstract

We propose InstructDET, a data-centric method for referring object detection (ROD) that localizes target objects based on user instructions. While deriving from referring expressions (REC), the instructions we leverage are greatly diversified to encompass common user intentions related to object detection. For one image, we produce tremendous instructions that refer to every single object and different combinations of multiple objects. Each instruction and its corresponding object bounding boxes (bbxs) constitute one training data pair. In order to encompass common detection expressions, we involve emerging vision-language model (VLM) and large language model (LLM) to generate instructions guided by text prompts and object bbxs, as the generalizations of foundation models are effective to produce human-like expressions (e.g., describing object property, category, and relationship). We name our constructed dataset as InDET. It contains images, bbxs and generalized instructions that are from foundation models. Our InDET is developed from existing REC datasets and object detection datasets, with the expanding potential that any image with object bbxs can be incorporated through using our InstructDET method. By using our InDET dataset, we show that a conventional ROD model surpasses existing methods on standard REC datasets and our InDET test set. Our data-centric method InstructDET, with automatic data expansion by leveraging foundation models, directs a promising field that ROD can be greatly diversified to execute common object detection instructions.

Figures (15)

• Figure 1: Our ROD aims to execute diversified user detection instructions compared to visual grounding. For images with object bbxs, we use foundation models to produce human-like object detection instructions. By training a conventional ROD model with incorporating tremendous instructions, we largely push ROD towards practical usage from a data-centric perspective.
• Figure 2: An overview of our InstructDET. We use two pipelines to produce detection expressions via foundation models. In the global prompt pipeline, we use LLaVA to describe an image via text, and combine this text with other text prompts for LLaMA input. In the local prompt pipeline, we use the same image with object bbxs and text prompts as multi modality input for LLaVA. The produced expressions are further refined to instructions and incorporated into our InDET dataset.
• Figure 3: Expression filtering by image visual prompting and visual-textual matching via CLIP.
• Figure 4: Mining commonalities among multi-objects via expression concatenation and text semantic clustering, followed by LLaMA descriptions on each cluster center.
• Figure 5: Dataset analysis of expression length, diversity and group distributions.