Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

InViC: Intent-aware Visual Cues for Medical Visual Question Answering

Zhisong Wang, Ziyang Chen, Zanting Ye, Hongze Zhu, Yefeng Zheng, Yong Xia

Abstract

Medical visual question answering (Med-VQA) aims to answer clinically relevant questions grounded in medical images. However, existing multimodal large language models (MLLMs) often exhibit shortcut answering, producing plausible responses by exploiting language priors or dataset biases while insufficiently attending to visual evidence. This behavior undermines clinical reliability, especially when subtle imaging findings are decisive. We propose a lightweight plug-in framework, termed Intent-aware Visual Cues (InViC), to explicitly enhance image-based answer generation in medical VQA. InViC introduces a Cue Tokens Extraction (CTE) module that distills dense visual tokens into a compact set of K question-conditioned cue tokens, which serve as structured visual intermediaries injected into the LLM decoder to promote intent-aligned visual evidence. To discourage bypassing of visual information, we further design a two-stage fine-tuning strategy with a cue-bottleneck attention mask. In Stage I, we employ an attention mask to block the LLM's direct view of raw visual features, thereby funneling all visual evidence through the cue pathway. In Stage II, standard causal attention is restored to train the LLM to jointly exploit the visual and cue tokens. We evaluate InViC on three public Med-VQA benchmarks (VQA-RAD, SLAKE, and ImageCLEF VQA-Med 2019) across multiple representative MLLMs. InViC consistently improves over zero-shot inference and standard LoRA fine-tuning, demonstrating that intent-aware visual cues with bottlenecked training is a practical and effective strategy for improving trustworthy Med-VQA.

InViC: Intent-aware Visual Cues for Medical Visual Question Answering

Abstract

Medical visual question answering (Med-VQA) aims to answer clinically relevant questions grounded in medical images. However, existing multimodal large language models (MLLMs) often exhibit shortcut answering, producing plausible responses by exploiting language priors or dataset biases while insufficiently attending to visual evidence. This behavior undermines clinical reliability, especially when subtle imaging findings are decisive. We propose a lightweight plug-in framework, termed Intent-aware Visual Cues (InViC), to explicitly enhance image-based answer generation in medical VQA. InViC introduces a Cue Tokens Extraction (CTE) module that distills dense visual tokens into a compact set of K question-conditioned cue tokens, which serve as structured visual intermediaries injected into the LLM decoder to promote intent-aligned visual evidence. To discourage bypassing of visual information, we further design a two-stage fine-tuning strategy with a cue-bottleneck attention mask. In Stage I, we employ an attention mask to block the LLM's direct view of raw visual features, thereby funneling all visual evidence through the cue pathway. In Stage II, standard causal attention is restored to train the LLM to jointly exploit the visual and cue tokens. We evaluate InViC on three public Med-VQA benchmarks (VQA-RAD, SLAKE, and ImageCLEF VQA-Med 2019) across multiple representative MLLMs. InViC consistently improves over zero-shot inference and standard LoRA fine-tuning, demonstrating that intent-aware visual cues with bottlenecked training is a practical and effective strategy for improving trustworthy Med-VQA.
Paper Structure (20 sections, 6 equations, 2 figures, 2 tables)

This paper contains 20 sections, 6 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Overview
  4. CTE Module
  5. Question-Conditioned Slot Initialization.
  6. Cross-Modal Retriever.
  7. Residual Cue Calibration with Scaled Gating.
  8. Two-Stage Training with Cue Bottleneck
  9. Stage I: Cue-Bottleneck Training.
  10. Stage II: Full-Context Refinement.
  11. Training Objective
  12. Experiments and Results
  13. Datasets and Evaluation Metrics
  14. Datasets.
  15. Evaluation Metrics.
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Overview of InViC. Left: CTE module with learnable cue slots initialized from a pooled question representation, followed by cross-attention to question tokens, cross-attention to visual tokens, and adapter-style calibration. Middle: Two-stage training strategy where visual tokens are visible only to cue tokens in Stage I and fully visible in Stage II. Right: Sequence construction and LLM training with cross-entropy loss.
  • Figure 2: Qualitative comparison of Lingshu-7B and Qwen3-VL-4B on the SLAKE dataset. We visualize model predictions after zero-shot inference, LoRA fine-tuning, and integration of the InViC architecture. Blue highlights indicate correct predictions, while red indicates incorrect predictions.