When Gradient Optimization Is Not Enough: $\dagger$ Dispersive and Anchoring Geometric Regularizer for Multimodal Learning

TL;DR

This work shows that gradient optimization alone can fail to produce well-conditioned multimodal representations, manifesting intra-modal collapse and cross-modal drift. It introduces DAGR, a plug-and-play geometry regularizer operating on normalized embeddings with two components: intra-modal dispersion to maximize diversity and inter-modal anchoring to bound cross-modal drift without forcing global alignment. The authors provide theoretical guarantees linking dispersion to entropy and effective rank, and demonstrate a maximum-entropy interpretation under a drift budget. Empirically, DAGR yields consistent gains in both multimodal fusion and unimodal robustness across diverse benchmarks (CREMA-D, Kinetics-Sounds, CUBICC, XRF55) and remains lightweight with adaptive Pareto balancing to reduce hyperparameter tuning. The results suggest that shaping representation geometry is a powerful, practical approach to mitigating modality trade-offs in multimodal learning and can be extended to broader modalities and tasks.

Abstract

Multimodal learning aims to integrate complementary information from heterogeneous modalities, yet strong optimization alone does not guaranty well-structured representations. Even under carefully balanced training schemes, multimodal models often exhibit geometric pathologies, including intra-modal representation collapse and sample-level cross-modal inconsistency, which degrade both unimodal robustness and multimodal fusion. We identify representation geometry as a missing control axis in multimodal learning and propose \regName, a lightweight geometry-aware regularization framework. \regName enforces two complementary constraints on intermediate embeddings: an intra-modal dispersive regularization that promotes representation diversity, and an inter-modal anchoring regularization that bounds sample-level cross-modal drift without rigid alignment. The proposed regularizer is plug-and-play, requires no architectural modifications, and is compatible with various training paradigms. Extensive experiments across multiple multimodal benchmarks demonstrate consistent improvements in both multimodal and unimodal performance, showing that explicitly regulating representation geometry effectively mitigates modality trade-offs.

Figures (12)

• Figure 1: Progressive geometric alignment in multimodal representation learning.(Left)Modality-dominated geometry: embeddings are primarily organized by modality, leading to compact but isolated modality-specific clusters and poor cross-modal semantic alignment. (Middle)Regularized dispersion and anchoring: intra-modal dispersion encourages samples within each modality to spread out, while inter-modal anchoring pulls semantically corresponding samples across modalities closer, progressively reshaping the representation geometry. (Right)Semantically aligned geometry: embeddings corresponding to the same semantic concept across different modalities form coherent, modality-invariant clusters, yielding a well-aligned multimodal representation space.
• Figure 2: Training-time geometric diagnostics. Left: semantic margin $\Delta_{\mathrm{sem}}$ (Eq. \ref{['eq:delta_sem_def']}), where DAGR steadily improves semantic separability compared with both the baseline and Disp Only. Middle: effective rank of unimodal representations, showing that dispersion increases intrinsic dimensionality. Right: cross-modal deviation converging to a non-zero plateau, consistent with an irreducible modality gap; DAGR achieves more stable alignment than dispersion alone.
• Figure 3: Cross-modal similarity geometry. (a) Cosine similarity distributions between positive (matched) and negative (mismatched) cross-modal pairs under the DGL baseline. (b) The corresponding distributions after adding a dispersive loss with an alignment/anchoring component, showing increased separation (larger $\Delta\mu$ and $D_{\mathrm{KS}}$). (c) Retrieval performance measured by Recall@K, where improved separability translates into higher recall.
• Figure 4: t-SNE visualization of multimodal embeddings on CREMA-D. DAGR produces more compact and better-aligned semantic clusters compared with the baseline.
• Figure 5: t-SNE visualization of multimodal embeddings on CUBICC. DAGR improves semantic compactness and stabilizes image--caption alignment relative to the baseline.

Theorems & Definitions (12)

• Lemma 3.1: Repulsive gradients of dispersive potentials
• Theorem 3.2
• Theorem 3.3
• Remark 3.4
• proof
• proof
• proof
• proof
• Corollary 1.1
• proof