CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning

TL;DR

This work tackles cross-level semantic asynchrony in multimodal learning by structuring each modality into a three-level hierarchy and restricting cross-modal exchange to a learned, level-aligned shared subspace with token budgets. It introduces Intra-Level Co-Exchange Domain (IntraCED) for level-wise shared/private decomposition and budgeted cross-attention, and Inter-Level Co-Aggregation Domain (InterCAD) for anchor-guided cross-level synchronization and modality selection, coupled with intra- and inter-level regularizers. The overall objective combines a task loss with $\\mathcal L_{ ext{Intra}}$ and $\\mathcal L_{ ext{Inter}}$, enforcing separation and reducing cross-level interference. Experiments on six benchmarks spanning emotion recognition, event localization, sentiment analysis, and action recognition show that CLCR achieves strong, generalizable performance and provides interpretable insights into modality contributions and level-wise importance. This framework offers a principled approach to mitigate semantic misalignment in hierarchical multimodal data, with potential impact on robust, real-world multimodal systems.

Abstract

Multimodal learning aims to capture both shared and private information from multiple modalities. However, existing methods that project all modalities into a single latent space for fusion often overlook the asynchronous, multi-level semantic structure of multimodal data. This oversight induces semantic misalignment and error propagation, thereby degrading representation quality. To address this issue, we propose Cross-Level Co-Representation (CLCR), which explicitly organizes each modality's features into a three-level semantic hierarchy and specifies level-wise constraints for cross-modal interactions. First, a semantic hierarchy encoder aligns shallow, mid, and deep features across modalities, establishing a common basis for interaction. And then, at each level, an Intra-Level Co-Exchange Domain (IntraCED) factorizes features into shared and private subspaces and restricts cross-modal attention to the shared subspace via a learnable token budget. This design ensures that only shared semantics are exchanged and prevents leakage from private channels. To integrate information across levels, the Inter-Level Co-Aggregation Domain (InterCAD) synchronizes semantic scales using learned anchors, selectively fuses the shared representations, and gates private cues to form a compact task representation. We further introduce regularization terms to enforce separation of shared and private features and to minimize cross-level interference. Experiments on six benchmarks spanning emotion recognition, event localization, sentiment analysis, and action recognition show that CLCR achieves strong performance and generalizes well across tasks.

Figures (8)

• Figure 1: Cross-level semantic asynchrony and CLCR. Top: mixing across levels causes semantic confusion and mismatch. bottom: CLCR structures each modality into three aligned levels and restricts exchange to the learned shared subspace at matched levels, enabling level semantic alignment and reliable aggregation.
• Figure 2: Overview of the proposed CLCR framework. Each modality is organized into a three-level semantic hierarchy. IntraCED performs budgeted cross-modal exchange in an explicitly disentangled level-shared subspace, while InterCAD synchronizes and aggregates shared and private streams across levels into the final task representation.
• Figure 3: Qualitative analysis: t-SNE of MOSI representations. CLCR yields more compact, better separated clusters than ablated variants, indicating improved semantic alignment.
• Figure 4: Visualization of semantic-level importance and learned weights on MOSI and KS. The shallow, mid, and deep levels are used in a complementary rather than redundant way, supporting the necessity of the semantic hierarchy.
• Figure 5: Robustness analysis under Gaussian noise. performance curves over more than 10 random runs with different noisy data on MOSI (a) and KS (b), $\downarrow$ indicates that lower values are better.