NeIn: Telling What You Don't Want

TL;DR

The paper tackles the underexplored problem of negation understanding in vision-language models for text-guided image editing. It introduces NeIn, a large-scale dataset of 366,957 quintuplets built via a generation-and-filtering pipeline that leverages BLIP for object detection, InstructPix2Pix fine-tuned on MagicBrush for editing, and LLaVA-NeXT for quality filtering, to create challenging negative-instruction samples and corresponding target images. An evaluation protocol combining removal and retention metrics with both VQA and open-vocabulary detection tools demonstrates that current state-of-the-art image-editing models struggle with negative prompts, though fine-tuning on NeIn yields notable improvements. This work highlights negation as a critical, yet inadequately addressed, aspect of aligning vision-language models with human information needs and provides a foundation for extending negation understanding to broader vision-language tasks.

Abstract

Negation is a fundamental linguistic concept used by humans to convey information that they do not desire. Despite this, minimal research has focused on negation within text-guided image editing. This lack of research means that vision-language models (VLMs) for image editing may struggle to understand negation, implying that they struggle to provide accurate results. One barrier to achieving human-level intelligence is the lack of a standard collection by which research into negation can be evaluated. This paper presents the first large-scale dataset, Negative Instruction (NeIn), for studying negation within instruction-based image editing. Our dataset comprises 366,957 quintuplets, i.e., source image, original caption, selected object, negative sentence, and target image in total, including 342,775 queries for training and 24,182 queries for benchmarking image editing methods. Specifically, we automatically generate NeIn based on a large, existing vision-language dataset, MS-COCO, via two steps: generation and filtering. During the generation phase, we leverage two VLMs, BLIP and InstructPix2Pix (fine-tuned on MagicBrush dataset), to generate NeIn's samples and the negative clauses that expresses the content of the source image. In the subsequent filtering phase, we apply BLIP and LLaVA-NeXT to remove erroneous samples. Additionally, we introduce an evaluation protocol to assess the negation understanding for image editing models. Extensive experiments using our dataset across multiple VLMs for text-guided image editing demonstrate that even recent state-of-the-art VLMs struggle to understand negative queries.

Figures (8)

• Figure 1: The failures of recent text-guided image editing methods in understanding the negative queries.
• Figure 2: The process to create our dataset. It consists of two main steps: generation and filtering. ITM and VQA are image-text matching and visual question answering, respectively.
• Figure 3: Illustration for fine-tuning and benchmarking process.
• Figure 4: Qualitative results of five SOTA methods on NeIn’s evaluation samples (first two samples) and random image-prompt pairs (last two samples). The fine-tuned InstructPix2Pix ($3^{\text{rd}}$ column) and MagicBrush ($5^{\text{th}}$ column) on NeIn's training set are foolightorange highlighted.
• Figure 5: Statistical analysis for NeIn. a) Number of samples after each filtering phase; b) Number of instances per pre-defined format for $\mathcal{T}_{n}$, the x-axis is followed to the order presented Table 2; c) Number of instances per object category, these 80 objects follow the objects in MS-COCO dataset. Best view in zoom.