Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Latent Representation Tuning for Image-text Classification

Hao Sun, Yu Song

TL;DR

The paper tackles robust multimodal image–text classification when one modality may be missing by freezing image and text foundation models and introducing Modality Latent Translation (MoLT), which per layer projects embeddings into a shared latent space and uses cross-attention plus $L_{CCA}$ to promote cross-modal alignment before applying $FBP$ to form robust representations. A subsequent fusion stage aggregates per-layer robust features with modality embeddings to produce predictions, guided by a joint objective $L = eta L_{CE} + eta L_{CCA}$ that balances task performance and cross-modal correlation. Evaluations on MM-IMDB, UPMC-Food101, and SNLI-VE demonstrate state-of-the-art results and strong robustness to modality absence and noise, highlighting the practical value of maintaining frozen pretrained foundations while enabling robust cross-modal reasoning. Overall, the approach advances robust multimodal understanding in realistic settings where modalities can be missing or degraded, with broad implications for deployment in vision–language tasks.

Abstract

Large models have demonstrated exceptional generalization capabilities in computer vision and natural language processing. Recent efforts have focused on enhancing these models with multimodal processing abilities. However, addressing the challenges posed by scenarios where one modality is absent remains a significant hurdle. In response to this issue, we propose a robust latent representation tuning method for large models. Specifically, our approach introduces a modality latent translation module to maximize the correlation between modalities, resulting in a robust representation. Following this, a newly designed fusion module is employed to facilitate information interaction between the modalities. Within this framework, common semantics are refined during training, and robust performance is achieved even in the absence of one modality. Importantly, our method maintains the frozen state of the image and text foundation models to preserve their capabilities acquired through large-scale pretraining. We conduct experiments on several public datasets, and the results underscore the effectiveness of our proposed method.

Robust Latent Representation Tuning for Image-text Classification

TL;DR

The paper tackles robust multimodal image–text classification when one modality may be missing by freezing image and text foundation models and introducing Modality Latent Translation (MoLT), which per layer projects embeddings into a shared latent space and uses cross-attention plus to promote cross-modal alignment before applying to form robust representations. A subsequent fusion stage aggregates per-layer robust features with modality embeddings to produce predictions, guided by a joint objective that balances task performance and cross-modal correlation. Evaluations on MM-IMDB, UPMC-Food101, and SNLI-VE demonstrate state-of-the-art results and strong robustness to modality absence and noise, highlighting the practical value of maintaining frozen pretrained foundations while enabling robust cross-modal reasoning. Overall, the approach advances robust multimodal understanding in realistic settings where modalities can be missing or degraded, with broad implications for deployment in vision–language tasks.

Abstract

Large models have demonstrated exceptional generalization capabilities in computer vision and natural language processing. Recent efforts have focused on enhancing these models with multimodal processing abilities. However, addressing the challenges posed by scenarios where one modality is absent remains a significant hurdle. In response to this issue, we propose a robust latent representation tuning method for large models. Specifically, our approach introduces a modality latent translation module to maximize the correlation between modalities, resulting in a robust representation. Following this, a newly designed fusion module is employed to facilitate information interaction between the modalities. Within this framework, common semantics are refined during training, and robust performance is achieved even in the absence of one modality. Importantly, our method maintains the frozen state of the image and text foundation models to preserve their capabilities acquired through large-scale pretraining. We conduct experiments on several public datasets, and the results underscore the effectiveness of our proposed method.
Paper Structure (14 sections, 9 equations, 2 figures, 3 tables)

This paper contains 14 sections, 9 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Large Vision and Language Models
  4. Multimodal Large Model Tuning
  5. Method
  6. Modality Latent Translation
  7. Fusion and Training Target
  8. Experiments and Analysis
  9. Benchmark Datasets
  10. Experimental Settings
  11. Quantitative Results
  12. Ablation Study
  13. Robust Inference Analysis
  14. Conclusion

Figures (2)

  • Figure 1: The overview of our proposed method. The image and text are first processed by separate encoders for robust representation learning. During this process, the robust representation $H_r$ is obtained by MolT module and FBP. After that we fuse the modality features and robust embedding for the final predictions.
  • Figure 2: The visualization of some cases for our propose method and the baseline model. The examples are from the SNLI-VE dataset.