When Vision Models Meet Parameter Efficient Look-Aside Adapters Without Large-Scale Audio Pretraining

TL;DR

The paper tackles the data- and compute-intensive barrier of large-scale audio pretraining by enabling direct, one-stage fine-tuning of ImageNet-pretrained vision transformers for audio tasks using a Look-Aside Adapter (LoAA). LoAA introduces time–frequency aware interactions through 1D convolutional projections inside adapters, providing a modality-specific inductive bias while keeping the backbone frozen. Across EPIC-SOUNDS, ESC-50, and Speech Commands, LoAA matches or exceeds dedicated audio-pretrained baselines, notably surpassing them on EPIC-SOUNDS at modest parameter budgets. This work offers a resource-efficient, effective approach for cross-modal audio understanding and highlights the value of tailored adapter designs in bridging vision and audio modalities.

Abstract

Recent studies show that pretrained vision models can boost performance in audio downstream tasks. To enhance the performance further, an additional pretraining stage with large scale audio data is typically required to infuse audio specific knowledge into the vision model. However, such approaches require extensive audio data and a carefully designed objective function. In this work, we propose bypassing the pretraining stage by directly fine-tuning the vision model with our Look Aside Adapter (LoAA) designed for efficient audio understanding. Audio spectrum data is represented across two heterogeneous dimensions time and frequency and we refine adapters to facilitate interactions between tokens across these dimensions. Our experiments demonstrate that our adapters allow vision models to reach or surpass the performance of pretrained audio models in various audio and speech tasks, offering a resource efficient and effective solution for leveraging vision models in audio applications.

Figures (3)

• Figure 1: An illustration of our simplified approach for audio classification. Our newly proposed Parameter Efficient Fine-Tuning (PEFT) paradigm for audio classification is a direct adaptation to downstream tasks in a singular stage. This approach even outperforms the current paradigm, which involves pretraining with large-scale audio datasets such as AudioSet-2M, as evidenced by its performance on the EPIC-SOUNDS dataset.
• Figure 2: Graphical illustration of adapter module pipelines of conventional parallel adapter (left) and our look-aside adapter (right). $d$: input token dimension; $r$: bottleneck dimension; $T$: time sequence of each token.
• Figure 3: Attention maps of bell sound produced by AST-based models trained in different ways. The audio data is obtained from AudioSet and represented by 128 dimensions of frequency and 1024 timesteps. We examine the self-attention of tokens from the last layer, and display the results for the key timesteps of 320-336. Each token corresponds to $16\times16$ area of the audio spectrogram and the attention maps are made up of $8\times64$ tokens. For clearer visualization, the tokens are upscaled to their original size.