Improving Medical Visual Representation Learning with Pathological-level Cross-Modal Alignment and Correlation Exploration

TL;DR

This work tackles data scarcity and the complexity of long medical reports by introducing PLACE, a framework that enhances medical visual representation learning through pathological-level cross-modal alignment (PCMA) and a cross-modal correlation exploration (CCE). A Visual Pathology Observation Extractor (VPOE) with pathology query tokens yields Visual Pathology Observation Representations (V-PORs) that are paired with Textual Pathological Observations (T-PORs) for robust, pathology-focused alignment without disease labels. The CCE objective further enriches representations by predicting patch-level covariance conditioned on reports, promoting fine-grained locality-aware understanding. Empirically, PLACE achieves state-of-the-art performance across segmentation, detection, classification, retrieval, and report generation on standard medical datasets, with strong zero-shot capabilities and good performance in low-data settings; code is published for reproducibility.

Abstract

Learning medical visual representations from image-report pairs through joint learning has garnered increasing research attention due to its potential to alleviate the data scarcity problem in the medical domain. The primary challenges stem from the lengthy reports that feature complex discourse relations and semantic pathologies. Previous works have predominantly focused on instance-wise or token-wise cross-modal alignment, often neglecting the importance of pathological-level consistency. This paper presents a novel framework PLACE that promotes the Pathological-Level Alignment and enriches the fine-grained details via Correlation Exploration without additional human annotations. Specifically, we propose a novel pathological-level cross-modal alignment (PCMA) approach to maximize the consistency of pathology observations from both images and reports. To facilitate this, a Visual Pathology Observation Extractor is introduced to extract visual pathological observation representations from localized tokens. The PCMA module operates independently of any external disease annotations, enhancing the generalizability and robustness of our methods. Furthermore, we design a proxy task that enforces the model to identify correlations among image patches, thereby enriching the fine-grained details crucial for various downstream tasks. Experimental results demonstrate that our proposed framework achieves new state-of-the-art performance on multiple downstream tasks, including classification, image-to-text retrieval, semantic segmentation, object detection and report generation. Code is available at https://github.com/Markin-Wang/PLACE.

Figures (6)

• Figure 1: The illustration of (a) the token-wise alignment, (b) clustered-based pseudo-pathology level and (c) our proposed pathological-level alignment. V-PORs is the abbreviation of visual pathology observation representations which are associated with specific anatomical regions.
• Figure 2: The overall architecture of $PLACE$. Our model takes advantage of both the instance-level and pathological-level cross-modal alignments during the joint-training. The proposed Visual Pathology Observation Extractor utilizes the learnable pathology query tokens to derive the visual pathology observation representations which are then aligned with the textual pathology observation representations by the PCMA module. The cross-modal correlation exploration module calculates the covariance matrix for the image patches and requires the model to predict this matrix based on the global report representation. Through these carefully designed objectives, PLACE is capable of learning a more generalizable and fine-grained visual representation.
• Figure 3: T-SNE visualisation for the extracted VPORs from 100 randomly selected samples in the test set.
• Figure 4: An illustration of the pathological-level cross-modal alignment between the V-POR and T-POR. Each sentence (row) refers to a pathology observation from one sample. The proposed Visual Pathology Observation Extractor extracts a group of V-PORs for each image through the same learnable pathology query tokens.
• Figure 5: Visualization of the attention map of the V-PORs to the local visual tokens. These V-PORs are those having the highest similarity to the associated atelectasis-related T-PORs. The highlighted visual tokens are regarded as important regions learnt by the model.