Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning

TL;DR

This work proposes EGMA, a gaze-guided multi-modal framework for medical vision-language pre-training that leverages radiologists’ eye-gaze data to explicitly align image regions with diagnostic text. By introducing a fine-grained eye-gaze guided alignment and a gaze-informed cross-modality mapping, EGMA achieves state-of-the-art performance on image classification and image-text retrieval across multiple chest-imaging datasets, including strong zero-shot capabilities. Ablation studies demonstrate that even small amounts of gaze data can enhance generalization, and visualizations confirm improved localization of disease regions and tighter clustering of disease representations. The approach highlights the practical value and feasibility of incorporating gaze data into medical multi-modal learning, with implications for improved interpretability and data-efficient training.

Abstract

In the medical multi-modal frameworks, the alignment of cross-modality features presents a significant challenge. However, existing works have learned features that are implicitly aligned from the data, without considering the explicit relationships in the medical context. This data-reliance may lead to low generalization of the learned alignment relationships. In this work, we propose the Eye-gaze Guided Multi-modal Alignment (EGMA) framework to harness eye-gaze data for better alignment of medical visual and textual features. We explore the natural auxiliary role of radiologists' eye-gaze data in aligning medical images and text, and introduce a novel approach by using eye-gaze data, collected synchronously by radiologists during diagnostic evaluations. We conduct downstream tasks of image classification and image-text retrieval on four medical datasets, where EGMA achieved state-of-the-art performance and stronger generalization across different datasets. Additionally, we explore the impact of varying amounts of eye-gaze data on model performance, highlighting the feasibility and utility of integrating this auxiliary data into multi-modal alignment framework.

Figures (6)

• Figure 1: The guiding role of radiologists' eye-gaze data. The text provided by radiologists during diagnosis aligns naturally with the attention regions.
• Figure 2: The framework of EGMA. After images and text are processed by the encoder in Part $A$, patch feature and sentence feature representations are obtained, resulting in a fine-grained similarity matrix for instances. Subsequently, the two types of eye-gaze-based auxiliary information obtained in Part $B$ are used for fine-grained and cross-mapping alignment in Part $C$ and Part $D$, respectively.
• Figure 3: Results of cross-modality attention maps visualization. Related text content: (a) "heart size borderline enlarged"; (b) "increased bibasilar opacities are the combination of increased bilateral pleural effusions and bibasilar atelectasis".
• Figure 4: t-SNE visualization on CheXpert 5x200 dataset by CLIP and our EGMA. The figures display points of different colors representing various ground truth disease types and their cluster assignments. The color-coded points illustrate the clustering results of each algorithm.
• Figure 5: The generation methods for heatmap at both word-level and sentence-level.