Image Difference Grounding with Natural Language

TL;DR

This work defines Image Difference Grounding (IDG), a task for precisely localizing cross-image visual differences guided by natural language. It introduces DiffGround, a large-scale dataset with dual-image edits and instruction-bound queries, and DiffTracker, a DETR-based baseline featuring a Difference Enhancement Module to emphasize inter-image differences while suppressing shared content. The method combines a Differentiation Vision Backbone, a multimodal neck, and an advanced transformer decoder to produce bounding boxes for user-indicated differences, achieving strong gains over existing visual grounding models on DiffGround. The dataset and model are released publicly to spur further research in fine-grained, cross-modal perception and practical multimodal systems for real-world applications.

Abstract

Visual grounding (VG) typically focuses on locating regions of interest within an image using natural language, and most existing VG methods are limited to single-image interpretations. This limits their applicability in real-world scenarios like automatic surveillance, where detecting subtle but meaningful visual differences across multiple images is crucial. Besides, previous work on image difference understanding (IDU) has either focused on detecting all change regions without cross-modal text guidance, or on providing coarse-grained descriptions of differences. Therefore, to push towards finer-grained vision-language perception, we propose Image Difference Grounding (IDG), a task designed to precisely localize visual differences based on user instructions. We introduce DiffGround, a large-scale and high-quality dataset for IDG, containing image pairs with diverse visual variations along with instructions querying fine-grained differences. Besides, we present a baseline model for IDG, DiffTracker, which effectively integrates feature differential enhancement and common suppression to precisely locate differences. Experiments on the DiffGround dataset highlight the importance of our IDG dataset in enabling finer-grained IDU. To foster future research, both DiffGround data and DiffTracker model will be publicly released.

Figures (7)

• Figure 1: Task Comparison between classic visual grounding (VG), visual change detection (VCD), image difference captioning (IDC), and our proposed image difference grounding (IDG) for finer-grained vision-language understanding.
• Figure 2: The illustration of our data engine for building the DiffGround dataset.
• Figure 3: Our DiffGround dataset statistics for each difference pattern. (a), (b) and (c) respective illustrates the statistics of the word diversity for each object category (i.e., first row) and the occurrence frequency of different categories (i.e., second row) within a single difference pattern (i.e., coarse-grained object category change, fine-grained object appearance change and direct object removal). The horizontal coordinates for (a), (b), (c) are the examples of the specific categories with the ranked top 50 highest vertical values.
• Figure 4: DiffGround dataset statistics. (a) the number of IDC instructions per object's category (i.e., ranked top 8K) in the log scale. (b) the word cloud highlights the head categories.
• Figure 5: Visualizations of samples from our DiffGround dataset.