Text-centric Alignment for Multi-Modality Learning

TL;DR

This work tackles modality mismatch in multimodal learning by proposing TAMML, a text-centered framework that converts all modalities to text and uses LLMs with in-context learning for text-style translation, modality summarization, and reasoning augmentation. By treating text as a unified semantic space, TAMML enables zero-shot generalization to unseen modality combinations and reduces reliance on retraining for new modalities. Empirical results on real datasets show TAMML outperforms embedding-based baselines across various modality pairs and LLM configurations, with notable gains when using larger models and robust ablations highlighting the contribution of each component. The approach offers a flexible, scalable pathway for real-world multimodal systems where modality availability is dynamic and uncertain, with potential impact on fields from healthcare to finance.

Abstract

This research paper addresses the challenge of modality mismatch in multimodal learning, where the modalities available during inference differ from those available at training. We propose the Text-centric Alignment for Multi-Modality Learning (TAMML) approach, an innovative method that utilizes Large Language Models (LLMs) with in-context learning and foundation models to enhance the generalizability of multimodal systems under these conditions. By leveraging the unique properties of text as a unified semantic space, TAMML demonstrates significant improvements in handling unseen, diverse, and unpredictable modality combinations. TAMML not only adapts to varying modalities but also maintains robust performance, showcasing the potential of foundation models in overcoming the limitations of traditional fixed-modality frameworks in embedding representations. This study contributes to the field by offering a flexible, effective solution for real-world applications where modality availability is dynamic and uncertain.

Figures (7)

• Figure 1: This paper establishes a general method for all mismatch types and combinations. As the figure shows, the model utilizes three modalities during training. Our unified model handles inference for any combination of modalities, such as User1’s unseen audio-video combination or the diverse combinations presented by User2 and User3.
• Figure 2: Traditional downstream training relies on embeddings extracted from upstream foundation models, with one foundation model designated for each modality. This approach limits the downstream model's ability to adapt to unseen modalities at test time without undergoing complete retraining. Previous research has addressed this issue by implementing zero-shot cross-modality translations during the inference phase.
• Figure 3: In the training phase, each raw input modality is transformed into text representations using a corresponding foundation model. Following the modality transformation, summarization, and augmentation are applied in parallel. Finally, the output texts are concatenated as the training inputs to a transformer model for downstream prediction. The inference phase follows a similar pattern, with the exception of utilizing an LLM for the text-style translation after the text transformation module. We apply a one-shot in-context learning approach to adapt the linguistic style as anticipated during training.
• Figure 4: Examples of prompt templates for each modules
• Figure 5: The left and right pictures illustrate the visualizations of embeddings for image and text data, respectively, before and after our processes.