Multimodal Deep Learning for Low-Resource Settings: A Vector Embedding Alignment Approach for Healthcare Applications

TL;DR

This work tackles the resource constraints of deploying multimodal deep learning in healthcare within LMICs by evaluating vector embeddings extracted from foundation models as a lightweight alternative to end-to-end raw-data processing. It systematically compares single-modal embeddings and Vision-Language Model embeddings against raw data across three medical datasets, using early and late fusion schemes, and introduces an embedding alignment method to close modality gaps amidst the cone effect. Results show embeddings dramatically reduce memory and compute while achieving competitive accuracy and F1 scores, with alignment providing additional performance boosts. The findings advocate for sustainable AI practices by enabling efficient multimodal inference and training in low-resource settings, while acknowledging domain-specific limitations and the need for adaptive, task-aware embedding strategies.

Abstract

Large-scale multi-modal deep learning models have revolutionized domains such as healthcare, highlighting the importance of computational power. However, in resource-constrained regions like Low and Middle-Income Countries (LMICs), limited access to GPUs and data poses significant challenges, often leaving CPUs as the sole resource. To address this, we advocate for leveraging vector embeddings to enable flexible and efficient computational methodologies, democratizing multimodal deep learning across diverse contexts. Our paper investigates the efficiency and effectiveness of using vector embeddings from single-modal foundation models and multi-modal Vision-Language Models (VLMs) for multimodal deep learning in low-resource environments, particularly in healthcare. Additionally, we propose a simple yet effective inference-time method to enhance performance by aligning image-text embeddings. Comparing these approaches with traditional methods, we assess their impact on computational efficiency and model performance using metrics like accuracy, F1-score, inference time, training time, and memory usage across three medical modalities: BRSET (ophthalmology), HAM10000 (dermatology), and SatelliteBench (public health). Our findings show that embeddings reduce computational demands without compromising model performance. Furthermore, our alignment method improves performance in medical tasks. This research promotes sustainable AI practices by optimizing resources in constrained environments, highlighting the potential of embedding-based approaches for efficient multimodal learning. Vector embeddings democratize multimodal deep learning in LMICs, particularly in healthcare, enhancing AI adaptability in varied use cases.

Figures (6)

• Figure 1: Schematic Representation of Multimodal Fusion Approaches. (1A) depicts the traditional multimodal fusion approach using raw data. The approach processes text and images through BERT and ViT models respectively. (1B) shows our embedding multimodal modeling approach, illustrating the extraction of image and text embeddings from foundation models and their subsequent utilization in multimodal learning. (1C) shows the two distinct approaches for data fusion. C-1 represents an early fusion approach, where embeddings are concatenated at the input and passed through a feature extraction block, followed by a classification layer. C-2 presents the late-joint fusion approach, highlighting the separate feature extraction from each modality and their integration at a later stage.
• Figure 2: Embedding modality gap between image and text embeddings for medical and non-medical datasets. (A) Represents the medical image (orange), and text (blue) embeddings generated using CLIP. (B) Represents the general image (orange), and text (blue) embeddings generated using CLIP from non-medical benchmark datasets. The embedding representations were normalized to fit inside on a unit sphere, and PCA method was used to reduce the dimensionality for visualization.
• Figure 3: Embedding alignment in the medical datasets represented as image embeddings (orange), text embeddings (blue). 3A shows the original embedding representation of each dataset with no shift. 3B Shows the embedding alignment process pooling together both embedding modalities into the same space.
• Figure 4: Metrics calculated over shifts from negative shift -1, to positive shift 1 for BRSET Dataset.
• Figure 5: Metrics calculated over shifts from negative shift -1, to positive shift 1 for HAM 10000 Dataset.