MoLT: Mixture of Layer-Wise Tokens for Efficient Audio-Visual Learning

TL;DR

MoLT tackles the computational bottleneck of adapting large audio-visual transformers by distilling latent tokens only from late layers through a mixture of uni-modal and cross-modal adapters. A router selects adapter outputs to form layer-wise tokens, which are fused by a Token Fusion Module with weights that reflect their relative importance; token orthogonality regularization further ensures token diversity. Across AVQA, AVS, and AVE, MoLT achieves state-of-the-art results while drastically reducing trainable parameters and memory usage. The approach is validated through extensive ablations, showing that late-layer adaptation, adapter composition, and TOR are crucial for effective cross-modal reasoning under constrained resources.

Abstract

In this paper, we propose Mixture of Layer-Wise Tokens (MoLT), a parameter- and memory-efficient adaptation framework for audio-visual learning. The key idea of MoLT is to replace conventional, computationally heavy sequential adaptation at every transformer layer with a parallel, lightweight scheme that extracts and fuses layer-wise tokens only from the late layers. We adopt two types of adapters to distill modality-specific information and cross-modal interaction into compact latent tokens in a layer-wise manner. A token fusion module then dynamically fuses these layer-wise tokens by taking into account their relative significance. To prevent the redundancy of latent tokens, we apply an orthogonality regularization between latent tokens during training. Through the systematic analysis of the position of adaptation in the pre-trained transformers, we extract latent tokens only from the late layers of the transformers. This strategic adaptation approach avoids error propagation from the volatile early-layer features, thereby maximizing the adaptation performance while maintaining parameter and memory efficiency. Through extensive experiments, we demonstrate that MoLT outperforms existing methods on diverse audio-visual benchmarks, including Audio-Visual Question Answering, Audio-Visual Segmentation, and Audio-Visual Event Localization.

Figures (5)

• Figure 1: The proposed MoLT achieves the state-of-the-art performance on various audio-visual downstream tasks, while ensuring efficiency in terms of both parameter and memory.
• Figure 2: Overview of MoLT framework. The proposed framework consists of three components: (1) pre-trained transformer models, (2) Feature Distillation Module that distills cross-modal interactions and modality-specific information from each pre-trained layer into layer-wise latent tokens, and (3) Token Fusion Module that assigns importance weights to these tokens and fuses them accordingly.
• Figure 3: Details of (a) Uni-modal Distillation Adapter, and (b) Cross-modal Distillation Adapter.
• Figure 4: Cosine similarity between input and output of each transformer layer for audio and visual branches on the AVE task.
• Figure 5: Qualitative results to show the effect of token orthogonality regularization. (a) Cosine similarity map between layer-wise latent tokens, (b) weights for each layer-wise latent token produced by the token fusion module.