Asymmetric Hierarchical Anchoring for Audio-Visual Joint Representation: Resolving Information Allocation Ambiguity for Robust Cross-Modal Generalization

TL;DR

The paper tackles information allocation ambiguity in symmetric, discrete audio-visual representations used for Cross-Modal Generalization (CMG). It introduces Asymmetric Hierarchical Anchoring (AHA), which anchors semantic content in an audio-driven RVQ hierarchy and distills video semantics into a shared discrete space, aided by a GRL-based adversarial decoupler and Local Sliding Alignment for fine-grained temporal alignment. Core contributions include the asymmetric RVQ anchor mechanism, a robust GRL-based disentanglement strategy, Local Sliding Alignment, and the integration of Cross-CPC with MM-EMA to stabilize codebooks. Empirically, AHA achieves state-of-the-art cross-modal transfer on AVE and AVVP benchmarks and demonstrates improved semantic disentanglement in a Talking Face Disentanglement setup, suggesting broad applicability to controllable multimodal tasks.

Abstract

Audio-visual joint representation learning under Cross-Modal Generalization (CMG) aims to transfer knowledge from a labeled source modality to an unlabeled target modality through a unified discrete representation space. Existing symmetric frameworks often suffer from information allocation ambiguity, where the absence of structural inductive bias leads to semantic-specific leakage across modalities. We propose Asymmetric Hierarchical Anchoring (AHA), which enforces directional information allocation by designating a structured semantic anchor within a shared hierarchy. In our instantiation, we exploit the hierarchical discrete representations induced by audio Residual Vector Quantization (RVQ) to guide video feature distillation into a shared semantic space. To ensure representational purity, we replace fragile mutual information estimators with a GRL-based adversarial decoupler that explicitly suppresses semantic leakage in modality-specific branches, and introduce Local Sliding Alignment (LSA) to encourage fine-grained temporal alignment across modalities. Extensive experiments on AVE and AVVP benchmarks demonstrate that AHA consistently outperforms symmetric baselines in cross-modal transfer. Additional analyses on talking-face disentanglement experiment further validate that the learned representations exhibit improved semantic consistency and disentanglement, indicating the broader applicability of the proposed framework.

Figures (11)

• Figure 1: The Cross-Modal Generalization (CMG) Task Definition
• Figure 2: Illustration of the challenges and goals in audio-visual joint representation learning. (a) Modality Gap: Audio (blue) and video (red) features are separated in the latent space due to modality-specific biases. This is why we need to separate the semantic and specific parts; (b) Bad Unified Representation: A naive unification leads to information leakage and semantic codebook collapse, where semantic and modality-specific factors are entangled around discrete codes; (c) Good Unified Representation: A well-structured unified space aligns cross-modal semantic features around shared discrete representations while isolating modality-specific variations.
• Figure 3: Overview of the proposed AHA architecture. (a) The main pipeline of our entire model. Through our experiments, we found that the effective semantic information in the audio RVQ is generally concentrated in the first two layers. Therefore, we make slight adjustments to k according to the information density required by the task. For the CMG task, to unify the comparison with previous state-of-the-art work, we set k=1 by default. For the subsequent Talking Face Disentanglement Experiments \ref{['fig:face']}, we set k=2; (b) Framework of Local Sliding Alignment (LSA); (c) Framework of Gradient Reversal Layer(GRL)-based Adversarial Decoupler
• Figure 4: The impact of GRL's adversarial intensity (larger $\lambda_{max}$ indicates stronger adversarial effect), RVQ's number of extra audio layers and Codebook size on experimental accuracy.
• Figure 5: (a) Talking Face Disentanglement Experiment Results: We selected two videos from sources not in the training set to demonstrate the effect. To better showcase the experimental results, we selected frames where Specific Video and Semantic Video have significant differences (for example: whether eyes are closed and mouth is open) for comparison. We set the identity information as the corresponding identity vector from the Semantic Video. For comparison, we also tested (i) removing GRL from our asymmetric structure (w/o $L_{\text{grl}}$). (ii) replacing GRL with CLUB in the asymmetric structure($L_{\text{grl}} \rightarrow L_{\text{CLUB}}$). (iii) adding GRL to a symmetric structure identical to Unicode (Symmetric). (Please see the \ref{['TalkingFaceExperiment']} for more experimental results); (b) PCA & UMAP Results: For the same selected video features, we extracted specific features and semantic features, and performed Principal Component Analysis (PCA) and Uniform Manifold Approximation and Projection (UMAP)