Efficient Domain Adaptation of Multimodal Embeddings using Constrastive Learning

TL;DR

This work addresses the challenge of deploying powerful multimodal foundation-model embeddings in resource-limited domains, such as healthcare, where fine-tuning large models is often impractical. It introduces a lightweight, task-specific embedding adaptor: a small nonlinear projection trained with a contrastive objective on frozen embeddings, enabling improved downstream classification without updating the base models. Empirical results on healthcare notes and a multimodal movie dataset show significant gains over unprojected and PCA baselines, with per-modality projections frequently delivering the largest improvements, while maintaining minimal computational overhead. The approach is modality-agnostic and CPU-friendly, making advanced foundation-model capabilities accessible in real-world, resource-constrained environments and offering a scalable pathway for domain adaptation across diverse data types.

Abstract

Recent advancements in machine learning (ML), natural language processing (NLP), and foundational models have shown promise for real-life applications in critical, albeit compute-constrainted fields like healthcare. In such areas, combining foundational models with supervised ML offers potential for automating tasks like diagnosis and treatment planning, but the limited availability of onsite computational resources pose significant challenges before applying these technologies effectively: Current approaches either yield subpar results when using pretrained models without task-specific adaptation, or require substantial computational resources for fine-tuning, which is often a barrier to entry in such environments. This renders them inaccessible in applications where performance and quality standards are high, but computational resources are scarce. To bridge the gap between best-in-class performance and accessibility, we propose a novel method for adapting foundational, multimodal embeddings to downstream tasks, without the need of expensive fine-tuning processes. Our method leverages frozen embeddings from Large Language Models (LLMs) and Vision Models, and uses contrastive learning to train a small, task-specific nonlinear projection that can be used in the downstream task, without having to fine-tune the original foundational models. We show that this efficient procedure leads to significant performance improvements across various downstream tasks, and perhaps more importantly with minimal computational overhead, offering a practical solution for the use of advanced, foundational ML models in resource-constrained settings.

Figures (5)

• Figure 1: Multimodal prediction with task-agnostic embeddings
• Figure 2: Multimodal prediction with task-specific embeddings
• Figure 3: Tabular movie information as a single text
• Figure 4: t-SNE plot of embeddings before and after contrastive projection.
• Figure 5: Time-performance trade-off