Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Shared Encoder Approach to Multimodal Representation Learning

Shuvendu Roy, Franklin Ogidi, Ali Etemad, Elham Dolatabadi, Arash Afkanpour

TL;DR

The paper tackles the challenge of scarce paired multimodal data in medical AI by proposing a fully shared encoder architecture that uses learnable modality embeddings to handle text and image inputs within a single Transformer stack. It adopts a contrastive learning objective inspired by CLIP and demonstrates that sharing parameters across modalities yields superior or competitive retrieval performance, particularly in low-data scenarios, compared to modality-specific baselines and larger pretrained models. Key contributions include systematic analysis of modality information integration (embeddings vs tokens) and the effect of early versus late modality-specific layers, plus evidence that parameter sharing improves data efficiency. The work suggests a practical path toward robust, data-efficient multimodal medical AI with reduced reliance on large proprietary encoders, and provides code to facilitate reproducibility and further research.

Abstract

Multimodal representation learning has demonstrated remarkable potential in enabling models to process and integrate diverse data modalities, such as text and images, for improved understanding and performance. While the medical domain can benefit significantly from this paradigm, the scarcity of paired multimodal data and reliance on proprietary or pretrained encoders pose significant challenges. In this work, we present a shared encoder framework for multimodal representation learning tailored to the medical domain. Our approach employs a single set of encoder parameters shared across modalities, augmented with learnable modality features. Empirical results demonstrate that our shared encoder idea achieves superior performance compared to separate modality-specific encoders, demonstrating improved generalization in data-constrained settings. Notably, the performance gains are more pronounced with fewer training examples, underscoring the efficiency of our shared encoder framework for real-world medical applications with limited data. Our code and experiment setup are available at https://github.com/VectorInstitute/shared_encoder.

A Shared Encoder Approach to Multimodal Representation Learning

TL;DR

The paper tackles the challenge of scarce paired multimodal data in medical AI by proposing a fully shared encoder architecture that uses learnable modality embeddings to handle text and image inputs within a single Transformer stack. It adopts a contrastive learning objective inspired by CLIP and demonstrates that sharing parameters across modalities yields superior or competitive retrieval performance, particularly in low-data scenarios, compared to modality-specific baselines and larger pretrained models. Key contributions include systematic analysis of modality information integration (embeddings vs tokens) and the effect of early versus late modality-specific layers, plus evidence that parameter sharing improves data efficiency. The work suggests a practical path toward robust, data-efficient multimodal medical AI with reduced reliance on large proprietary encoders, and provides code to facilitate reproducibility and further research.

Abstract

Multimodal representation learning has demonstrated remarkable potential in enabling models to process and integrate diverse data modalities, such as text and images, for improved understanding and performance. While the medical domain can benefit significantly from this paradigm, the scarcity of paired multimodal data and reliance on proprietary or pretrained encoders pose significant challenges. In this work, we present a shared encoder framework for multimodal representation learning tailored to the medical domain. Our approach employs a single set of encoder parameters shared across modalities, augmented with learnable modality features. Empirical results demonstrate that our shared encoder idea achieves superior performance compared to separate modality-specific encoders, demonstrating improved generalization in data-constrained settings. Notably, the performance gains are more pronounced with fewer training examples, underscoring the efficiency of our shared encoder framework for real-world medical applications with limited data. Our code and experiment setup are available at https://github.com/VectorInstitute/shared_encoder.

Paper Structure

This paper contains 15 sections, 3 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminaries
  5. Shared Encoder Framework
  6. Modality Identifier.
  7. Shared Encoder.
  8. Layer Sharing and Modality-Specific Layers
  9. Experiments
  10. Setup
  11. Results
  12. Comparing Shared vs Modality-Specific Encoders.
  13. Impact of Parameter Sharing Across Training Set Sizes.
  14. Integrating Modality-Specific Layers with Shared Encoders.
  15. Conclusion

Figures (3)

  • Figure 1: Top: Encoding data with (left) individual encoders vs (right) a shared encoder. Bottom: Adding modality-specific information by (left) appending modality embeddings to feature vectors, vs (right) adding a modality token.
  • Figure 2: Comparison of methods for incorporating modality information into a shared encoder. The baseline (dark dashed line) uses two disjoint modality-specific encoders (125M parameters total), while the Large CLIP model (dotted line) has 210M parameters. All shared encoders have 125M parameters.
  • Figure 3: Impact of adding modality-specific layers to encoders. The dashed line shows the performance of a purely shared encoder, with green arrows indicating performance gains from early modality layers and orange arrows showing gains (or losses) from late modality layers.