To Align or Not to Align: Strategic Multimodal Representation Alignment for Optimal Performance

TL;DR

The paper investigates when explicitly aligning multimodal representations helps or hinders unimodal encoders. It introduces a controllable cross-modal contrastive loss parameterized by $\lambda$ and leverages Partial Information Decomposition (PID) to characterize redundancy, uniqueness, and synergy across modalities. Through controlled synthetic data and three real-world benchmarks (AV-MNIST, CMU-MOSEI, MUStARD), it shows that alignment boosts performance in redundancy-dominant settings, harms performance when modality-specific information dominates, and yields mixed gains in synergy-dominant contexts with an identifiable optimal $\lambda^*$. The work offers practical guidance for alignment-aware multimodal learning by tailoring alignment strength to the underlying information structure of the data.

Abstract

Multimodal learning often relies on aligning representations across modalities to enable effective information integration, an approach traditionally assumed to be universally beneficial. However, prior research has primarily taken an observational approach, examining naturally occurring alignment in multimodal data and exploring its correlation with model performance, without systematically studying the direct effects of explicitly enforced alignment between representations of different modalities. In this work, we investigate how explicit alignment influences both model performance and representation alignment under different modality-specific information structures. Specifically, we introduce a controllable contrastive learning module that enables precise manipulation of alignment strength during training, allowing us to explore when explicit alignment improves or hinders performance. Our results on synthetic and real datasets under different data characteristics show that the impact of explicit alignment on the performance of unimodal models is related to the characteristics of the data: the optimal level of alignment depends on the amount of redundancy between the different modalities. We identify an optimal alignment strength that balances modality-specific signals and shared redundancy in the mixed information distributions. This work provides practical guidance on when and how explicit alignment should be applied to achieve optimal unimodal encoder performance.

Figures (7)

• Figure 1: Overview of our experimental framework for analyzing the impact of explicit multimodal alignment. ① Independent Training Baseline: Each unimodal encoder is trained independently to establish a performance and representation baseline, capturing naturally emergent alignment. ② Controlled Cross-modal Alignment: We introduce a contrastive learning module to explicitly enforce alignment during training, enabling systematic analysis of how alignment strength affects unimodal performance and representational similarity. ③ Real-world Generalization via PID: We apply Partial Information Decomposition (PID) to quantify the redundancy–uniqueness–synergy structure across modality pairs, validating our findings under diverse real-world data conditions.
• Figure 2: Synthetic Data Generation Process.
• Figure 3: Experimental results on the synthetic dataset across varying redundancy levels (R). (a)–(e): Classification performance of Encoder 1 (blue) and Encoder 2 (red) as the alignment strength $\lambda$ increases, under different redundancy levels ($R = 0, 2, 4, 6, 8$). (f)–(j): Cross-modal representation alignment scores (CKA, SVCCA, and Mutual-KNN) between the two encoders as a function of $\lambda$, evaluated at the same redundancy levels.
• Figure 4: Effect of alignment strength $\lambda$ on synthetic-dataset encoder1 performance under varying redundancy.
• Figure 5: Performance and representation alignment as a function of alignment strength $\lambda$ across different real-world datasets. (a)–(b): Results on AV-MNIST. (c)–(d): Results on MUStARD (V–A pair). (e)–(f): Results on CMU-MOSEI (V–T pair).