Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classification

TL;DR

This work tackles the data- and compute-constrained setting of applying foundation-model pretraining to low-resource modalities by enabling cross-modal knowledge transfer through attention merging. The authors propose Multimodal Attention Merging (MAM), which transfers attention from high-resource modalities (text and vision) to speech/audio by interpolating Transformer Q/K/V projections via a fixed $\lambda$ (Attention Interpolation) or selectively merging layers, and a Learnable Interpolation variant $\lambda_i$ learned during fine-tuning. They demonstrate zero-shot improvements on ASR (HuBERT+BERT) and AEC (BEATs+ViT), with relative WER reductions up to 6.70% on LJ Speech, 1.80% on VCTK, and 10.63% reduction on ESC-50, and further gains when data/compute are available through Learnable-MAM (2.70%/2.90% WER reductions, 18.42% ESC-50). The results indicate transferable attention structure across modalities and offer a practical path to leverage abundant textual/visual pretraining for speech/audio tasks, with potential for scaling to larger models and multi-task setups.

Abstract

Training large foundation models using self-supervised objectives on unlabeled data, followed by fine-tuning on downstream tasks, has emerged as a standard procedure. Unfortunately, the efficacy of this approach is often constrained by both limited fine-tuning compute and scarcity in labeled downstream data. We introduce Multimodal Attention Merging (MAM), an attempt that facilitates direct knowledge transfer from attention matrices of models rooted in high resource modalities, text and images, to those in resource-constrained domains, speech and audio, employing a zero-shot paradigm. MAM reduces the relative Word Error Rate (WER) of an Automatic Speech Recognition (ASR) model by up to 6.70%, and relative classification error of an Audio Event Classification (AEC) model by 10.63%. In cases where some data/compute is available, we present Learnable-MAM, a data-driven approach to merging attention matrices, resulting in a further 2.90% relative reduction in WER for ASR and 18.42% relative reduction in AEC compared to fine-tuning.

Figures (1)

• Figure 1: Learnable Multimodal Attention Merging. The merged model correctly transcribes breech. modules are fine-tuned (trainable) while are frozen (non-trainable). In our initial approach, Multimodal Attention Merging, the interpolation factor $\lambda$ is frozen (non-trainable).