Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering

Yuanhao Zou, Zhaozheng Yin

TL;DR

AMiF tackles Med-VQA by unifying heterogeneous modality alignments across inter- and intra-modality signals and leveraging multi-view medical data. It introduces soft-label-based global alignment, OT-driven local alignment, hard negative mining across modalities, and a gated knowledge fusion decoder that selectively incorporates an answer vocabulary during fine-tuning. Empirically, AMiF achieves state-of-the-art performance on RAD-VQA, SLAKE, PathVQA, and VQA-2019, with ablations highlighting the contribution of unified alignment, hard-negative discrimination, and gating. The framework advances Med-VQA by robustly aligning multi-view medical data, effectively distinguishing hard negatives from positives, and filtering irrelevant knowledge for open-form clinical QA.

Abstract

Medical Visual Question Answering (Med-VQA) is a challenging task that requires a deep understanding of both medical images and textual questions. Although recent works leveraging Medical Vision-Language Pre-training (Med-VLP) have shown strong performance on the Med-VQA task, there is still no unified solution for modality alignment, and the issue of hard negatives remains under-explored. Additionally, commonly used knowledge fusion techniques for Med-VQA may introduce irrelevant information. In this work, we propose a framework to address these challenges through three key contributions: (1) a unified solution for heterogeneous modality alignments across multiple levels, modalities, views, and stages, leveraging methods like contrastive learning and optimal transport theory; (2) a hard negative mining method that employs soft labels for multi-modality alignments and enforces the hard negative pair discrimination; and (3) a Gated Cross-Attention Module for Med-VQA that integrates the answer vocabulary as prior knowledge and selects relevant information from it. Our framework outperforms the previous state-of-the-art on widely used Med-VQA datasets like RAD-VQA, SLAKE, PathVQA and VQA-2019.

Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering

TL;DR

AMiF tackles Med-VQA by unifying heterogeneous modality alignments across inter- and intra-modality signals and leveraging multi-view medical data. It introduces soft-label-based global alignment, OT-driven local alignment, hard negative mining across modalities, and a gated knowledge fusion decoder that selectively incorporates an answer vocabulary during fine-tuning. Empirically, AMiF achieves state-of-the-art performance on RAD-VQA, SLAKE, PathVQA, and VQA-2019, with ablations highlighting the contribution of unified alignment, hard-negative discrimination, and gating. The framework advances Med-VQA by robustly aligning multi-view medical data, effectively distinguishing hard negatives from positives, and filtering irrelevant knowledge for open-form clinical QA.

Abstract

Medical Visual Question Answering (Med-VQA) is a challenging task that requires a deep understanding of both medical images and textual questions. Although recent works leveraging Medical Vision-Language Pre-training (Med-VLP) have shown strong performance on the Med-VQA task, there is still no unified solution for modality alignment, and the issue of hard negatives remains under-explored. Additionally, commonly used knowledge fusion techniques for Med-VQA may introduce irrelevant information. In this work, we propose a framework to address these challenges through three key contributions: (1) a unified solution for heterogeneous modality alignments across multiple levels, modalities, views, and stages, leveraging methods like contrastive learning and optimal transport theory; (2) a hard negative mining method that employs soft labels for multi-modality alignments and enforces the hard negative pair discrimination; and (3) a Gated Cross-Attention Module for Med-VQA that integrates the answer vocabulary as prior knowledge and selects relevant information from it. Our framework outperforms the previous state-of-the-art on widely used Med-VQA datasets like RAD-VQA, SLAKE, PathVQA and VQA-2019.

Paper Structure

This paper contains 34 sections, 16 equations, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Modality Alignment for Med-VLP
  4. Hard Negatives for Med-VLP
  5. Med-VQA and Knowledge Fusion
  6. Methodology
  7. Overview
  8. Unified Modality Alignment
  9. Global-Level Alignment
  10. Inter-Modality (Image-Text)
  11. Intra-Modality (Multi-View & Text-Text)
  12. Soft Label Generation
  13. Alignment with Soft Labels
  14. Local-Level Alignment
  15. Hard Negative Mining
  16. ...and 19 more sections

Figures (6)

  • Figure 1: Motivations on: (a) unified modality alignment, (b) hard negative mining, and (c) selective knowledge fusion for Med-VQA.
  • Figure 2: Architecture of AMiF: (a) Pre-Training Phase, (b) Fine-Tuning Phase, and (c) Modality Co-Attention Module.
  • Figure 3: Architecture of Gated Knowledge Fusion Decoder, which consists of a Gated Cross-Attention Module and an Auto-Regressive Decoder.
  • Figure 4: Visualization of the local-level alignment related to underlined words for the ablative study of hard negative mining.
  • Figure 5: Attention map related to underlined words for the ablative study of selective knowledge fusion.
  • ...and 1 more figures