From Gaze to Insight: Bridging Human Visual Attention and Vision Language Model Explanation for Weakly-Supervised Medical Image Segmentation

TL;DR

This work tackles weakly supervised medical image segmentation by unifying clinician gaze with vision-language model (VLM) explanations in a teacher–student framework. The teacher combines high-confidence gaze with VLM-derived textual embeddings via a multi-scale cross-modal fusion and is supervised by a partial cross-entropy loss on precise gaze regions; knowledge is then distilled to a student trained on broader, noisier gaze data using feature-level distillation and confidence-aware consistency, reinforced by disagreement-aware masking. Key innovations include feature-level visual–text alignment, a confidence-aware consistency mechanism, and a disagreement-aware masking strategy to mitigate label noise, all demonstrated across Kvasir-SEG, NCI-ISBI, and ISIC with 3–5% Dice gains over gaze baselines and substantially reduced annotation burden. The method also preserves interpretability by maintaining gaze–text–prediction correlations, advancing deployable, annotation-efficient medical AI systems while highlighting avenues for uncertainty quantification and broader modality integration.

Abstract

Medical image segmentation remains challenging due to the high cost of pixel-level annotations for training. In the context of weak supervision, clinician gaze data captures regions of diagnostic interest; however, its sparsity limits its use for segmentation. In contrast, vision-language models (VLMs) provide semantic context through textual descriptions but lack the explanation precision required. Recognizing that neither source alone suffices, we propose a teacher-student framework that integrates both gaze and language supervision, leveraging their complementary strengths. Our key insight is that gaze data indicates where clinicians focus during diagnosis, while VLMs explain why those regions are significant. To implement this, the teacher model first learns from gaze points enhanced by VLM-generated descriptions of lesion morphology, establishing a foundation for guiding the student model. The teacher then directs the student through three strategies: (1) Multi-scale feature alignment to fuse visual cues with textual semantics; (2) Confidence-weighted consistency constraints to focus on reliable predictions; (3) Adaptive masking to limit error propagation in uncertain areas. Experiments on the Kvasir-SEG, NCI-ISBI, and ISIC datasets show that our method achieves Dice scores of 80.78%, 80.53%, and 84.22%, respectively-improving 3-5% over gaze baselines without increasing the annotation burden. By preserving correlations among predictions, gaze data, and lesion descriptions, our framework also maintains clinical interpretability. This work illustrates how integrating human visual attention with AI-generated semantic context can effectively overcome the limitations of individual weak supervision signals, thereby advancing the development of deployable, annotation-efficient medical AI systems. Code is available at: https://github.com/jingkunchen/FGI.

Figures (4)

• Figure 1: Schematic of the proposed gaze- and language-guided teacher–student framework. A VLM generates a structured explanation (location, boundary, characteristics, area, confidence) whose text embedding is fused with image features in the Teacher via Cross-Modal Attention (Sec. III-A). Gaze points are smoothed/thresholded to produce a high-confidence mask $M_{hc}$ and a broad-coverage mask $M_{bc}$ (Sec. III-A). Teacher$\rightarrow$Student feature-level distillation operates at stages $k{=}1\ldots4$, with the Angular Feature Consistency loss $\mathcal{L}_{AFC}$ aligning intermediate representations. Supervision uses $\mathcal{L}_{pCE}(M_{hc})$ and $\mathcal{L}_{CE}(M_{bc})$ (with DARM applied to $M_{bc}$ prior to $\mathcal{L}_{CE}$). The Confidence-aware Weighted Consistency loss$\mathcal{L}_{\mathrm{CWC}} =\mathcal{L}^{\mathrm{pos}}_{\mathrm{cwc}}+\mathcal{L}^{\mathrm{neg}}_{\mathrm{cwc}}$ is computed on the class-wise logits over $(\Omega_{\mathrm{pos}},\Omega_{\mathrm{neg}})$.
• Figure 2: Overview of the losses used in our framework. Left: Angular Feature Consistency ($\mathcal{L}_{\mathrm{AFC}}$) aligns intermediate features between the teacher and the student. Middle: Confidence-aware Weighted Consistency ($\mathcal{L}_{\mathrm{CWC}}$) regularizes predictions on confident positive/negative regions. Right: Supervision terms---the teacher uses partial cross-entropy ($\mathcal{L}_{\mathrm{pCE}}$) on high-confidence mask ($\mathbf{M}_{\mathrm{hc}}$), while the student uses cross-entropy ($\mathcal{L}_{\mathrm{CE}}$) on broad-coverage masks ($\mathbf{M}_{\mathrm{bc}}$) with disagreement-aware random masking (DRAM).
• Figure 3: Schematic illustration of using a large vision–language model to generate fixed-format textual descriptions.
• Figure 4: Qualitative comparison on three benchmarks. Rows correspond to Kvasir-SEG, NCI-ISBI, and ISIC. Columns (left to right): Ground Truth (GT), Gaze Map (GM), UNet, TransUNet (TrUNet), nnUNet, GazeMedSeg (GMS), Ours, and the VLM-derived Description. Red overlays indicate gaze fixations and reference boundaries; predicted masks are rendered as semi-transparent overlays for visual comparison. Across representative cases, our method yields more complete lesion coverage and sharper boundary adherence while reducing spurious activations in distractor regions.