OneDiff: A Generalist Model for Image Difference Captioning

TL;DR

OneDiff tackles Image Difference Captioning by introducing a generalist LVLM that fuses a siamese image encoder with a Visual Delta Module to capture fine-grained changes between image pairs. It is trained in two stages—Coupled Sample Training for cross-modal alignment and multi-task instruction tuning on the DiffCap dataset, which combines real and synthetic IDC data—to achieve robust cross-domain performance. The DiffCap dataset addresses data scarcity and diversity by merging real-world pairs, open-domain real-image pairs with GPT-generated captions, and synthetic edits generated via InstructPix2Pix with GPT-created difference captions. Empirically, OneDiff outperforms state-of-the-art methods across Spot-the-Diff, Image-Editing-Request, and Birds-to-Words, with substantial CIDEr gains (up to 211% on Image-Editing-Request and up to 97% CIDEr-point improvements on average), demonstrating strong generalization without task-specific tuning. This work advances practical IDC by delivering a scalable, generalist architecture and a rich training corpus with broad potential applications in surveillance, medical imaging, and asset editing.

Abstract

In computer vision, Image Difference Captioning (IDC) is crucial for accurately describing variations between closely related images. Traditional IDC methods often rely on specialist models, which restrict their applicability across varied contexts. This paper introduces the OneDiff model, a novel generalist approach that utilizes a robust vision-language model architecture, integrating a siamese image encoder with a Visual Delta Module. This innovative configuration allows for the precise detection and articulation of fine-grained differences between image pairs. OneDiff is trained through a dual-phase strategy, encompassing Coupled Sample Training and multi-task learning across a diverse array of data types, supported by our newly developed DiffCap Dataset. This dataset merges real-world and synthetic data, enhancing the training process and bolstering the model's robustness. Extensive testing on diverse IDC benchmarks, such as Spot-the-Diff, Image-Editing-Request, and Birds-to-Words, shows that OneDiff consistently outperforms existing state-of-the-art models in accuracy and adaptability, achieving improvements of up to 97% CIDEr points in average. By setting a new benchmark in IDC, OneDiff paves the way for more versatile and effective applications in detecting and describing visual differences. The code, models, and data will be made publicly available.

Figures (8)

• Figure 1: Illustration of Image Difference Captioning tasks.
• Figure 2: Data components and examples in the task-specific data collection of our DiffCap dataset.
• Figure 3: Automatic data construction pipeline of our DiffCap dataset.
• Figure 4: Architecture of our OneDiff model.
• Figure 5: Examples of OneDiff generated results.