What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging

TL;DR

This work addresses the critical gap in negation understanding within vision-language models by identifying an architectural affinity for neglecting negation cues and data scarcity of negation in training. It introduces CoVAND, a negation-focused, region-grounded dataset generated via three-step chain-of-thought captions and VQA-based alignment, and NegToMe, a semantic text token merging module with a negation boost that preserves polarity. Combined with a parameter-efficient LoRA fine-tuning approach, the method significantly improves negation grounding across multiple detectors and even generalizes to Multimodal Large Language Models, achieving substantial gains on D$^3$ and OVDEval, including the stricter NMS-AP metric. The work demonstrates strong qualitative and quantitative improvements, reduces false positives, and provides a practical path toward robust negation reasoning in real-world detection systems.

Abstract

State-of-the-art vision-language models (VLMs) suffer from a critical failure in understanding negation, often referred to as affirmative bias. This limitation is particularly severe in described object detection (DOD) tasks. To address this, we propose two primary contributions: (1) a new dataset pipeline and (2) a novel, lightweight adaptation recipe. First, we introduce CoVAND, a dataset constructed with a systematic chain-of-thought (CoT) and VQA-based pipeline to generate high-quality, instance-grounded negation data. Second, we propose NegToMe, a novel text token merging module that directly tackles the architectural cause of affirmative bias. NegToMe fundamentally addresses the structural loss of negation cues in tokenization, grouping them with attributes into coherent semantic phrases. It maintains correct polarity at the input level, enabling robust negation understanding even with limited data. For instance, to prevent a model from treating the fragmented tokens "not" and "girl" as simply "girl", NegToMe binds them into a single token whose meaning is correctly distinguished from that of "girl" alone. This module is integrated with a parameter-efficient and strategic LoRA fine-tuning approach. Our method significantly improves performance on challenging negation benchmarks with a lowered false positive rate, boosting NMS-AP by up to +10.8 points on OVDEval and demonstrating generalization to SoTA VLMs. This work marks a crucial step forward in addressing negation understanding for real-world detection applications.

Figures (25)

• Figure 1: Challenges with Negation Expressions. (a) Standard VLMs exhibit an affirmative bias, failing to distinguish contradictory negation queries. This issue stems from two causes: (b) the scarcity of negation words in standard datasets and (c) the model's tendency to assign low attention to negation cues. Our solutions, CoVAND and the NegToMe, directly address both problems.
• Figure 2: Dataset Generation Pipeline of the CoVAND. Our method first generates negation-focused captions for visually prompted regions using a three-step CoT process, then aligns each caption with the correct bounding box via VQA-based reasoning to ensure semantic correspondence.
• Figure 3: Overview of Training Pipeline. The input image and captions of CoVAND are encoded by frozen backbones. NegToMe assigns higher importance to negation cues in the text, and the LoRA adapter enables accurate localization of objects described by negated queries.
• Figure 4: Definition of Task and Metric.
• Figure 5: Dataset Statistics and Performance Scaling. (a) Statistics for our three CoVAND splits. (b) Bar plots with blue refer to NMS-AP and pink refer to FPR (lower is better).