Reframing Image Difference Captioning with BLIP2IDC and Synthetic Augmentation

TL;DR

This paper tackles Image Difference Captioning (IDC) in real-world settings by (i) adapting BLIP2 into a low-cost IDC model (BLIP2IDC) that encodes image differences via early attention on concatenated inputs and (ii) introducing Syned, a synthetic augmentation pipeline built on diffusion-based editing and LLM-ground-truth variation to enlarge IDC data. BLIP2IDC, fine-tuned with LoRA, outperforms existing two-stream IDC approaches on real-world benchmarks, and Syned further boosts CIDEr scores across models, establishing a new standard for IDC data quality and diversity. The combination of model adaptation and synthetic data yields improved generalization, particularly under distribution shifts, while acknowledging biases from pretraining, limitations of editing models, and potential LLM hallucinations. Overall, the work advances IDC by leveraging large-scale pretraining, end-to-end difference encoding, and scalable synthetic data to enable robust performance in real-world scenarios.

Abstract

The rise of the generative models quality during the past years enabled the generation of edited variations of images at an important scale. To counter the harmful effects of such technology, the Image Difference Captioning (IDC) task aims to describe the differences between two images. While this task is successfully handled for simple 3D rendered images, it struggles on real-world images. The reason is twofold: the training data-scarcity, and the difficulty to capture fine-grained differences between complex images. To address those issues, we propose in this paper a simple yet effective framework to both adapt existing image captioning models to the IDC task and augment IDC datasets. We introduce BLIP2IDC, an adaptation of BLIP2 to the IDC task at low computational cost, and show it outperforms two-streams approaches by a significant margin on real-world IDC datasets. We also propose to use synthetic augmentation to improve the performance of IDC models in an agnostic fashion. We show that our synthetic augmentation strategy provides high quality data, leading to a challenging new dataset well-suited for IDC named Syned1.

Figures (6)

• Figure 1: Results of state-of-the-art IDC models on: (a) a train sample of Emu Edit dataset, (b) a test sample and (c-d) zero-shot samples found in the wild. BLIP2IDC (Ours) is able to capture fine-grained differences, describe complex scenes, and generalize well to unseen data.
• Figure 2: Classic IDC pipeline. The feature extraction step is performed before training. The training step leverages the features from a frozen image encoder by concatenating them and then learns the difference representation from these concatenated features. The output is then fed to the text decoder to perform the generation of the difference caption in an autoregressive manner.
• Figure 3: BLIP2IDC end-to-end pipeline. We first resize and concatenate the image inputs before feeding them to the BLIP2 architecture fine-tuned using LoRA.
• Figure 4: Synthetic dataset creation pipeline leveraging a prompt-based image editing and large language models.
• Figure 5: Comparison of train set samples from EE, generated by the Emu Edit model, and Syned, generated by a fine-tuned InstructPix2Pix model.