Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OmniDiff: A Comprehensive Benchmark for Fine-grained Image Difference Captioning

Yuan Liu, Saihui Hou, Saijie Hou, Jiabao Du, Shibei Meng, Yongzhen Huang

TL;DR

OmniDiff addresses the IDC challenge by providing a breadth-depth benchmark built from real-world and 3D synthetic scenes with fine-grained, long captions. The authors propose M3Diff, a multimodal model augmented with a plug-and-play Multi-scale Differential Perception module that explicitly models cross-scale visual differences to enhance captioning accuracy. Across OmniDiff and public IDC benchmarks, M3Diff achieves state-of-the-art results, demonstrating strong cross-scenario generalization and robust fine-grained understanding. The work supplies the dataset, code, and models to advance IDC research and paves the way for extensions to continuous or video-based difference captioning.

Abstract

Image Difference Captioning (IDC) aims to generate natural language descriptions of subtle differences between image pairs, requiring both precise visual change localization and coherent semantic expression. Despite recent advancements, existing datasets often lack breadth and depth, limiting their applicability in complex and dynamic environments: (1) from a breadth perspective, current datasets are constrained to limited variations of objects in specific scenes, and (2) from a depth perspective, prior benchmarks often provide overly simplistic descriptions. To address these challenges, we introduce OmniDiff, a comprehensive dataset comprising 324 diverse scenarios-spanning real-world complex environments and 3D synthetic settings-with fine-grained human annotations averaging 60 words in length and covering 12 distinct change types. Building on this foundation, we propose M$^3$Diff, a MultiModal large language model enhanced by a plug-and-play Multi-scale Differential Perception (MDP) module. This module improves the model's ability to accurately identify and describe inter-image differences while maintaining the foundational model's generalization capabilities. With the addition of the OmniDiff dataset, M$^3$Diff achieves state-of-the-art performance across multiple benchmarks, including Spot-the-Diff, IEdit, CLEVR-Change, CLEVR-DC, and OmniDiff, demonstrating significant improvements in cross-scenario difference recognition accuracy compared to existing methods. The dataset, code, and models will be made publicly available to support further research.

OmniDiff: A Comprehensive Benchmark for Fine-grained Image Difference Captioning

TL;DR

OmniDiff addresses the IDC challenge by providing a breadth-depth benchmark built from real-world and 3D synthetic scenes with fine-grained, long captions. The authors propose M3Diff, a multimodal model augmented with a plug-and-play Multi-scale Differential Perception module that explicitly models cross-scale visual differences to enhance captioning accuracy. Across OmniDiff and public IDC benchmarks, M3Diff achieves state-of-the-art results, demonstrating strong cross-scenario generalization and robust fine-grained understanding. The work supplies the dataset, code, and models to advance IDC research and paves the way for extensions to continuous or video-based difference captioning.

Abstract

Image Difference Captioning (IDC) aims to generate natural language descriptions of subtle differences between image pairs, requiring both precise visual change localization and coherent semantic expression. Despite recent advancements, existing datasets often lack breadth and depth, limiting their applicability in complex and dynamic environments: (1) from a breadth perspective, current datasets are constrained to limited variations of objects in specific scenes, and (2) from a depth perspective, prior benchmarks often provide overly simplistic descriptions. To address these challenges, we introduce OmniDiff, a comprehensive dataset comprising 324 diverse scenarios-spanning real-world complex environments and 3D synthetic settings-with fine-grained human annotations averaging 60 words in length and covering 12 distinct change types. Building on this foundation, we propose MDiff, a MultiModal large language model enhanced by a plug-and-play Multi-scale Differential Perception (MDP) module. This module improves the model's ability to accurately identify and describe inter-image differences while maintaining the foundational model's generalization capabilities. With the addition of the OmniDiff dataset, MDiff achieves state-of-the-art performance across multiple benchmarks, including Spot-the-Diff, IEdit, CLEVR-Change, CLEVR-DC, and OmniDiff, demonstrating significant improvements in cross-scenario difference recognition accuracy compared to existing methods. The dataset, code, and models will be made publicly available to support further research.

Paper Structure

This paper contains 29 sections, 3 equations, 7 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Image Difference Captioning
  4. Multimodal Large Language Models
  5. The OmniDiff Dataset
  6. Overview of OmniDiff
  7. Dataset Construction Process
  8. Change Pair Collection.
  9. Difference Caption Collection.
  10. Dataset Statistics
  11. Evaluation protocol
  12. Comparisons with Existing Datasets
  13. Method
  14. Overview
  15. Multi-Scale Differential Perception
  16. ...and 14 more sections

Figures (7)

  • Figure 1: OmniDiff: a comprehensive benchmark comprising 324 complex real-world and 3D synthetic environments, featuring 12 distinct types of variations, with an average difference caption length of 60 words per image pair.
  • Figure 2: Overview of the OmniDiff construction pipeline.
  • Figure 3: The whole structure of M$^3$Diff with the multi-scale differential perception module.
  • Figure 4: Wordcloud visualization of the OmniDiff dataset.
  • Figure 5: Caption length distribution of the OmniDiff dataset.
  • ...and 2 more figures