Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bi-MCQ: Reformulating Vision-Language Alignment for Negation Understanding

Tae Hun Kim, Hyun Gyu Lee

TL;DR

The paper tackles the poor negation understanding of vision-language models in medical imaging, where InfoNCE-based contrastive learning yields weak discrimination between disease presence and absence. It introduces Bi-MCQ, a bidirectional benchmarked fine-tuning framework that trains Image-to-Text and Text-to-Image MCQ tasks using affirmative, negative, and mixed prompts, reframed as a conditional semantic comparison problem. A direction-specific Cross-Attention Fusion module decouples I2T and T2I alignment to reduce interference, with a loss defined as $L_{ ext{total}} = L_{ ext{I2T}} + L_{ ext{T2I}}$, enabling robust negation-sensitive reasoning. Across ChestXray14, Open-I, CheXpert, and PadChest, Bi-MCQ yields substantial gains in negation metrics (e.g., up to $0.47$ absolute AUC over zero-shot CARZero) and reduces the affirmative–negative AUC gap, demonstrating that objective reformulation can markedly enhance negation understanding in medical VLMs and generalize to other labeling-based vision–language tasks.

Abstract

Recent vision-language models (VLMs) achieve strong zero-shot performance via large-scale image-text pretraining and have been widely adopted in medical image analysis. However, existing VLMs remain notably weak at understanding negated clinical statements, largely due to contrastive alignment objectives that treat negation as a minor linguistic variation rather than a meaning-inverting operator. In multi-label settings, prompt-based InfoNCE fine-tuning further reinforces easy-positive image-prompt alignments, limiting effective learning of disease absence. To overcome these limitations, we reformulate vision-language alignment as a conditional semantic comparison problem, which is instantiated through a bi-directional multiple-choice learning framework(Bi-MCQ). By jointly training Image-to-Text and Text-to-Image MCQ tasks with affirmative, negative, and mixed prompts, our method implements fine-tuning as conditional semantic comparison instead of global similarity maximization. We further introduce direction-specific Cross-Attention fusion modules to address asymmetric cues required by bi-directional reasoning and reduce alignment interference. Experiments on ChestXray14, Open-I, CheXpert, and PadChest show that Bi-MCQ improves negation understanding by up to 0.47 AUC over the zero-shot performance of the state-of-the-art CARZero model, while achieving up to a 0.08 absolute gain on positive-negative combined (PNC) evaluation. Additionally, Bi-MCQ reduces the affirmative-negative AUC gap by an average of 0.12 compared to InfoNCE-based fine-tuning, demonstrating that objective reformulation can substantially enhance negation understanding in medical VLMs.

Bi-MCQ: Reformulating Vision-Language Alignment for Negation Understanding

TL;DR

The paper tackles the poor negation understanding of vision-language models in medical imaging, where InfoNCE-based contrastive learning yields weak discrimination between disease presence and absence. It introduces Bi-MCQ, a bidirectional benchmarked fine-tuning framework that trains Image-to-Text and Text-to-Image MCQ tasks using affirmative, negative, and mixed prompts, reframed as a conditional semantic comparison problem. A direction-specific Cross-Attention Fusion module decouples I2T and T2I alignment to reduce interference, with a loss defined as , enabling robust negation-sensitive reasoning. Across ChestXray14, Open-I, CheXpert, and PadChest, Bi-MCQ yields substantial gains in negation metrics (e.g., up to absolute AUC over zero-shot CARZero) and reduces the affirmative–negative AUC gap, demonstrating that objective reformulation can markedly enhance negation understanding in medical VLMs and generalize to other labeling-based vision–language tasks.

Abstract

Recent vision-language models (VLMs) achieve strong zero-shot performance via large-scale image-text pretraining and have been widely adopted in medical image analysis. However, existing VLMs remain notably weak at understanding negated clinical statements, largely due to contrastive alignment objectives that treat negation as a minor linguistic variation rather than a meaning-inverting operator. In multi-label settings, prompt-based InfoNCE fine-tuning further reinforces easy-positive image-prompt alignments, limiting effective learning of disease absence. To overcome these limitations, we reformulate vision-language alignment as a conditional semantic comparison problem, which is instantiated through a bi-directional multiple-choice learning framework(Bi-MCQ). By jointly training Image-to-Text and Text-to-Image MCQ tasks with affirmative, negative, and mixed prompts, our method implements fine-tuning as conditional semantic comparison instead of global similarity maximization. We further introduce direction-specific Cross-Attention fusion modules to address asymmetric cues required by bi-directional reasoning and reduce alignment interference. Experiments on ChestXray14, Open-I, CheXpert, and PadChest show that Bi-MCQ improves negation understanding by up to 0.47 AUC over the zero-shot performance of the state-of-the-art CARZero model, while achieving up to a 0.08 absolute gain on positive-negative combined (PNC) evaluation. Additionally, Bi-MCQ reduces the affirmative-negative AUC gap by an average of 0.12 compared to InfoNCE-based fine-tuning, demonstrating that objective reformulation can substantially enhance negation understanding in medical VLMs.
Paper Structure (22 sections, 9 equations, 4 figures, 3 tables)

This paper contains 22 sections, 9 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Chest X-ray-based Medical Vision–Language Models
  4. Negation Understanding in Vision–Language Models
  5. Method
  6. Limitations of Contrastive Fine-tuning for Negation Understanding
  7. Data Construction Process for Bi-MCQ-Fine-tuning
  8. Feature Extraction
  9. Image-to-Text Cross-Attention Alignment
  10. Text-to-Image Cross-Attention Alignment
  11. Bi-MCQ-Fine-Tuning Objective
  12. Experiments
  13. Datasets
  14. Evaluation Metrics
  15. Implementation Details
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Overview of the proposed Bi-MCQ-fine-tuning framework. Annotated chest X-ray data are organized into image-to-text (I2T) and text-to-image (T2I) MCQ batches and processed by image and text encoders. In I2T and T2I settings, modality-specific cross-attention is applied in each direction to obtain similarity scores $S^{\mathrm{I2T}}$ and $S^{\mathrm{T2I}}$, which are optimized using cross-entropy loss.
  • Figure 2: Data construction process for Bi-MCQ-fine-tuning. (a) Image-to-text (I2T) MCQ data construction, where positive, negative, and hybrid candidate prompts are generated for each image in a batch based on its annotation. (b) Text-to-image (T2I) MCQ data construction, where positive, negative, and hybrid query prompts are generated from batch-level annotations and corresponding candidate images are selected from the batch to form MCQ options.
  • Figure 3: Effect of fine-tuning strategies on CARZero across ChestXray14, Open-I, CheXpert, and PadChest datasets. For each method, the AUC obtained using positive prompts is visualized with higher opacity, while the AUC obtained using negative prompts is shown with lower opacity. The numeric values annotated above the bars indicate the absolute performance gap between positive and negative prompt AUCs, highlighting the robustness of each fine-tuning strategy to prompt polarity.
  • Figure 4: t-SNE visualization of feature embeddings of affirmative and negated prompts, highlighting differences in cluster structure associated with prompt polarity.