Enhancing medical vision-language contrastive learning via inter-matching relation modelling

TL;DR

This work addresses the limited ability of medical vision-language contrastive learning to differentiate semantically and clinically important local-image–text matchings. It introduces RECLF, a framework that jointly models semantic-relations (SRM) and importance-relations (IRM) among local-matchings to produce more fine-grained supervision for image representations. Through SRM and IRM, RECLF achieves superior performance across segmentation, linear/zero-shot classification, and cross-modal retrieval on multiple datasets, demonstrating enhanced generalization and localization of disease-related information. The results suggest that explicit inter-matching relation modelling improves the alignment between medical images and reports, enabling more accurate and robust downstream task performance.

Abstract

Medical image representations can be learned through medical vision-language contrastive learning (mVLCL) where medical imaging reports are used as weak supervision through image-text alignment. These learned image representations can be transferred to and benefit various downstream medical vision tasks such as disease classification and segmentation. Recent mVLCL methods attempt to align image sub-regions and the report keywords as local-matchings. However, these methods aggregate all local-matchings via simple pooling operations while ignoring the inherent relations between them. These methods therefore fail to reason between local-matchings that are semantically related, e.g., local-matchings that correspond to the disease word and the location word (semantic-relations), and also fail to differentiate such clinically important local-matchings from others that correspond to less meaningful words, e.g., conjunction words (importance-relations). Hence, we propose a mVLCL method that models the inter-matching relations between local-matchings via a relation-enhanced contrastive learning framework (RECLF). In RECLF, we introduce a semantic-relation reasoning module (SRM) and an importance-relation reasoning module (IRM) to enable more fine-grained report supervision for image representation learning. We evaluated our method using six public benchmark datasets on four downstream tasks, including segmentation, zero-shot classification, linear classification, and cross-modal retrieval. Our results demonstrated the superiority of our RECLF over the state-of-the-art mVLCL methods with consistent improvements across single-modal and cross-modal tasks. These results suggest that our RECLF, by modelling the inter-matching relations, can learn improved medical image representations with better generalization capabilities.

Figures (10)

• Figure 1: Illustration of commonly used mVLCL framework and our RECLF applied to a chest X-ray of a patient with a right basal effusion. One paired text sample (positive) and one unpaired text sample (hard negative) are shown as examples. (a) displays existing methods using a simple aggregation of local-matchings, and (b) displays our inter-matching relation-enhanced contrastive learning framework (RECLF). Triangles represent the local-matchings corresponding to the words on top and the image bounding box in same color. Black arrows show semantic-relations between local-matchings; green/red arrows show similarity variation of the local-matching after the semantic-relation reasoning module (SRM). The matrix $W$ represents the weight matrix of local-matchings of the importance-relation reasoning module (IRM).
• Figure 2: An overview of our RECLF. RECLF extracts representations of images and text and then calculates the similarities of the global- and local-matchings. SRM propagates the semantic information among all local-matchings and IRM aggregates all local-matchings with the importance guidance from the text encoder. Semantic contrastive loss is then applied to both the global and the local branches.
• Figure 3: t-SNE visualization of encoded global image representations on the MIMIC-5x200 dataset and the image encoders were initialized with different weights, including: (a) random weights, (b) ImageNet pretrained weights, (c) GLoRIA pretrained weights, (d) MGCA-ViT pretrained weights, and (e) our RECLF pretrained weights. Different colors indicate different class labels.
• Figure 4: Visualization of learned attention map for a given word. The input images are shown in the 1st row and the learned attention map are shown in the 2nd row. The attention values are normalized to 0-1, which was then mapped to the ‘jet’ lookup table. Red color represents the highest and blue color represents the lowest.
• Figure 5: An example text-to-image and image-to-text retrieval results of our method and the comparison MGCA method on the MIMIC-5x200 dataset. For each query, we present the top 5 retrieved results (from left to right). The associated labels are colored with green for a correct match (query and results are from the same patient), or with blue for semantic similar match (the results are describing the same disease but derived from different patients) or with red otherwise (the results are irrelevant). Only important sentences from the text report are reproduced here.