Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models

TL;DR

Img-Diff introduces a fully automated, contrastive data-synthesis pipeline to produce region-focused image-difference data for MLLMs. By generating highly similar image pairs with targeted object replacements, localizing difference regions with a Difference Area Generator, and producing precise, region-centric difference captions via a Difference Captions Generator, the authors create a high-quality object-replacement dataset. Fine-tuning LLaVA-1.5, MGM, and InternVL2 with Img-Diff yields substantial gains on image-difference benchmarks (e.g., MMVP, Spot-the-Diff, Image-Edit-Request) and across eight MLLM benchmarks, including surpassing GPT-4V and Gemini on MMVP. They demonstrate data quality and diversity, perform extensive ablations and supplementary analyses (including object-removal data), and open-source their dataset and tools to promote data-centric improvements in multimodal understanding.

Abstract

High-performance Multimodal Large Language Models (MLLMs) are heavily dependent on data quality. To advance fine-grained image recognition within MLLMs, we introduce a novel data synthesis method inspired by contrastive learning and image difference captioning. Our key idea involves challenging the model to discern both matching and distinct elements by scrutinizing object differences in detailed regions across similar images. We begin by generating pairs of similar images that emphasize object variations. Following this, we employ a Difference Area Generator to pinpoint object differences, and subsequently, a Difference Captions Generator to articulate these differences. This process results in a high-quality dataset of "object replacement" samples, termed Img-Diff, which can be scaled as needed due to its automated nature. We leverage this generated dataset to fine-tune state-of-the-art (SOTA) MLLMs, such as InternVL2, achieving substantial improvements across various image difference and Visual Question Answering tasks. Notably, the trained models significantly outperform existing SOTA models like GPT-4V and Gemini on the MMVP benchmark. Additionally, we conduct comprehensive evaluations to validate the dataset's diversity, quality, and robustness, offering several insights into the synthesis of such contrastive datasets. We release our codes and dataset to encourage further research on multimodal data synthesis and MLLMs' fundamental capabilities for image understanding.

Figures (10)

• Figure 1: Three "object replacement" examples within Img-Diff, highlighting fine-grined difference in both vision and language.
• Figure 2: The generation process for "object replacement" data.
• Figure 3: An overview of the Difference Area Generator and its three internal components: Image Similarity Filter, Image-text Matching Filter, and Difference Detector.
• Figure 4: An overview of the Difference Captions Generator and its two stages.
• Figure 5: Performance comparison on the MMVP benchmark.