TextME: Bridging Unseen Modalities Through Text Descriptions

TL;DR

TextME presents a text-only modality expansion framework that projects diverse, pretrained modality encoders into a unified LLM anchor space by exploiting a consistent modality gap. By centering modality and text embeddings and training lightweight projection networks using only unpaired text descriptions, TextME achieves zero-shot cross-modal transfer across six modalities with substantial performance preservation (average PPR around 74.5%). The method enables emergent cross-modal retrieval between modality pairs not seen during training and identifies geometric encoder properties (e.g., gap-content orthogonality variance) that predict success. Compared to paired-supervision approaches, TextME reduces data requirements by over 95% and avoids architectural coupling between encoders, offering a practical pathway to integrate specialized domains like medical imaging and molecular analysis into unified multimodal systems.

Abstract

Expanding multimodal representations to novel modalities is constrained by reliance on large-scale paired datasets (e.g., text-image, text-audio, text-3D, text-molecule), which are costly and often infeasible in domains requiring expert annotation such as medical imaging and molecular analysis. We introduce TextME, the first text-only modality expansion framework, to the best of our knowledge, projecting diverse modalities into LLM embedding space as a unified anchor. Our approach exploits the geometric structure of pretrained contrastive encoders to enable zero-shot cross-modal transfer using only text descriptions, without paired supervision. We empirically validate that such consistent modality gaps exist across image, video, audio, 3D, X-ray, and molecular domains, demonstrating that text-only training can preserve substantial performance of pretrained encoders. We further show that our framework enables emergent cross-modal retrieval between modality pairs not explicitly aligned during training (e.g., audio-to-image, 3D-to-image). These results establish text-only training as a practical alternative to paired supervision for modality expansion.

Figures (5)

• Figure 1: Comparison of modality expansion approaches. Unlike prior methods that require large-scale paired data or pseudo-pair construction through overlapping encoders, TextME achieves modality expansion using only unpaired text descriptions while reusing pretrained encoders.
• Figure 2: Overview of the TextME pipeline. (a) Offset computation estimates modality-specific centroids from unpaired samples, creating an interchangeable space where centered text and modal embeddings become functionally equivalent. (b) During training, projection networks are learned by aligning centered text embeddings with a unified LLM anchor space, requiring only text descriptions. (c) At inference, centering modal embeddings with the precomputed offset enables zero-shot cross-modal transfer without paired supervision.
• Figure 3: Semantic anchoring comparison. LLM embeddings and multimodal encoders are compared on 3K semantically equivalent cross-modal description pairs. LLM embeddings exhibit clearer separation between matched and unmatched pairs, demonstrating superior cross-domain alignment capability.
• Figure 4: Emergent cross-modal retrieval without paired supervision. Audio queries retrieve semantically related 3D objects (top), and molecular structures retrieve contextually appropriate images (bottom). These modality pairs were never seen during training, demonstrating that text-anchored alignment creates semantic bridges across arbitrary modalities.
• Figure 5: Orthogonality variance vs. performance preservation. Each point represents a single evaluation metric from six modalities, with tasks sharing the same encoder aligned vertically at identical variance values. Lower variance in gap-content orthogonality corresponds to higher downstream performance.