VISTA: Enhancing Vision-Text Alignment in MLLMs via Cross-Modal Mutual Information Maximization

TL;DR

This paper analyzes the cross-entropy objective in multimodal large language models (MLLMs) through an information-theoretic lens, showing that implicit cross-modal alignment diminishes as text length grows due to the linear growth of textual entropy while visual mutual information is bounded. To address this, it introduces Vision-Text Alignment (VISTA), a lightweight, parameter-free regularizer that explicitly maximizes cross-modal mutual information by incorporating a weighted MI term into the loss, using a stable proxy based on the final visual summary and a linearly increasing weight $f(t)$. The authors provide theoretical justification and practical instantiations, demonstrating that $f(t)$ mitigates alignment degradation and that VISTA improves performance across more than a dozen benchmarks on two backbones (TinyLLaVA and LLaVA-v1.5) without extra data or modules. Empirically, VISTA yields consistent gains in high-level VQA, general multimodal reasoning, and fine-grained perception, with MI-based (particularly L2) estimators often outperforming cosine similarity, highlighting the method’s robustness and scalability for improving cross-modal alignment in MLLMs.

Abstract

Current multimodal large language models (MLLMs) face a critical challenge in modality alignment, often exhibiting a bias towards textual information at the expense of other modalities like vision. This paper conducts a systematic information-theoretic analysis of the widely used cross-entropy loss in MLLMs, uncovering its implicit alignment objective. Our theoretical investigation reveals that this implicit objective has inherent limitations, leading to a degradation of cross-modal alignment as text sequence length increases, thereby hindering effective multimodal information fusion. To overcome these drawbacks, we propose Vision-Text Alignment (VISTA), a novel approach guided by our theoretical insights. VISTA introduces an explicit alignment objective designed to maximize cross-modal mutual information, preventing the degradation of visual alignment. Notably, VISTA enhances the visual understanding capabilities of existing MLLMs without requiring any additional trainable modules or extra training data, making it both efficient and practical. Our method significantly outperforms baseline models across more than a dozen benchmark datasets, including VQAv2, MMStar, and MME, paving the way for new directions in MLLM modal alignment research.

Figures (5)

• Figure 1: VISTA framework. We prevent target degradation caused by implicit alignment of the target by introducing the explicit alignment loss $\mathcal{L_{\text{VISTA}}}$of the text token and vision hidden state.
• Figure 2: Origin
• Figure 3: Vanilla LLaVA
• Figure 4: LLaVA-VISTA
• Figure 6: Semantic similarity visualization between MLP-generated prompt embeddings and image embeddings on LLaVA Wild Benchllava. Each subfigure presents two images per case: left—LLaVA baseline; right—our VISTA method. Darker colors indicate higher semantic alignment.

Theorems & Definitions (9)

• Lemma 3.1: Autoregressive entropy growth
• proof
• Lemma 3.2: Bounded visual mutual information
• proof
• Theorem 3.3: Vanishing vision-text alignment contribution
• proof
• proof
• proof
• proof