Representation-Regularized Convolutional Audio Transformer for Audio Understanding

TL;DR

The paper tackles bootstrap-based self-supervised learning for audio, which suffers from single-scale processing and slow convergence when learning from scratch. It introduces the Convolutional Audio Transformer (CAT), which combines a Multi-resolution Block to capture hierarchical temporal-spectral structures with a Representation Regularization objective that aligns the student with frozen external encoders. The training objective integrates $L = L_p + \lambda_1 L_g + \lambda_2 L_r$, with $L_r = \| p_\delta(C_s^{(d)}) - T \|_2^2$ guiding the student using representations from encoders such as CLAP, Audio-MAE, or AST, and the teacher is updated via EMA. Empirically, CAT achieves new state-of-the-art results on AudioSet (50.2 mAP on AS-2M; 47.8–47.9 mAP on AS-20K) and ESC-50, while converging about 5× faster than prior bootstrap methods, demonstrating the value of combining multi-scale processing with external semantic guidance for efficient audio understanding.

Abstract

Bootstrap-based Self-Supervised Learning (SSL) has achieved remarkable progress in audio understanding. However, existing methods typically operate at a single level of granularity, limiting their ability to model the diverse temporal and spectral structures inherent in complex audio signals. Furthermore, bootstrapping representations from scratch is computationally expensive, often requiring extensive training to converge. In this work, we propose the Convolutional Audio Transformer (CAT), a unified framework designed to address these challenges. First, to capture hierarchical audio features, CAT incorporates a Multi-resolution Block that aggregates information across varying granularities. Second, to enhance training efficiency, we introduce a Representation Regularization objective. Drawing inspiration from generative modeling, this auxiliary task guides the student model by aligning its predictions with high-quality semantic representations from frozen, pre-trained external encoders. Experimental results demonstrate that CAT significantly outperforms baselines on audio understanding benchmarks. Notably, it achieves competitive performance on the AudioSet 20k dataset with 5 times faster convergence than existing methods. Codes and checkpoints will be released soon at https://github.com/realzhouchushu/CAT.

Figures (3)

• Figure 1: CAT Pre-training Architecture Overview. The model follows a student-teacher bootstrap paradigm. The Student Encoder processes a masked spectrogram, while the Teacher Encoder (updated via Exponential Moving Average, EMA) receives the unmasked input. The training objective is composed of three parts: (1) Patch-level Loss ($L_p$): The student projector predicts the teacher's latent representations for masked patches; (2) Global Loss ($L_g$): Aligns the global CLS token of the student with the teacher's aggregated features; (3) Representation Loss ($L_r$): A regularization term that aligns intermediate representations from the student encoder with high-quality features extracted from a frozen external audio encoder.
• Figure 2: Multi-Resolution Block Architecture. This diagram depicts the data flow in the teacher's Multi-Resolution Block. The student branch follows a nearly identical process, differing primarily by the introduction of an input mask that is element-wise multiplied with the features inside the Convolutional Block.
• Figure 3: CAT Pre-training Convergence Performance. "w/o MR" denotes the model using a vanilla patch embedding instead of the multi-resolution block, while "w/o RR" denotes the model without representation regularization. The x-axis indicates the pre-training step (showing only the early phase), while the y-axis shows the best mAP on AS-20K after supervised fine-tuning with the corresponding checkpoint. For ease of comparison, the best reported performances of EAT, BEATs and Audio-MAE models are indicated by horizontal dashed lines in the plot.