AaPE: Aliasing-aware Patch Embedding for Self-Supervised Audio Representation Learning

TL;DR

The paper tackles aliasing introduced by aggressive patch-based spectrogram downsampling in transformer-based audio SSL by introducing AaPE, a drop-in patch embedding augmented with aliasing-aware frequency features derived from a Structured Bilateral Laplace Unit (SBLU).AaPE uses a Lambda Encoder to predict input-dependent decay and frequency per subband, an Adaptive SBLU to extract aliasing-focused features, and a Patch Fusion module to combine them with standard patch tokens, all within a teacher-student masked modeling framework with a multi-mask strategy and a contrastive objective.Experiments on AudioSet pre-training followed by fine-tuning and linear probing across diverse audio tasks show state-of-the-art or competitive results on several benchmarks, with clear gains on frequency-structured tasks and robust interpretability of the adaptive aliasing analysis.Overall, AaPE provides a complementary architectural improvement to SSL that mitigates aliasing without discarding important high-frequency information, and can be combined with other SSL strategies for further gains.

Abstract

Transformer-based audio SSL (self-supervised learning) models often treat spectrograms as images, applying convolutional patchification with heavy temporal downsampling. This lowers the effective Nyquist frequency and introduces aliasing, while naïve low-pass filtering removes task-relevant high-frequency cues. In this study, we present Aliasing-aware Patch Embedding (AaPE), a drop-in patch stem that mitigates aliasing while preserving high-frequency information. AaPE augments standard patch tokens with features produced by a band-limited complex sinusoidal kernel using a two-sided exponential window that dynamically targets alias-prone bands. Frequency and decay parameters of the kernel are estimated from the input, enabling parallel, adaptive subband analysis whose outputs are fused with the standard patch tokens. AaPE integrates seamlessly into the masked teacher-student self-supervised learning. In addition, we combine a multi-mask strategy with a contrastive objective to enforce consistency across diverse mask patterns, stabilizing training. Pre-training on AudioSet followed by fine-tuning evaluation across diverse downstream benchmarks, which spanned categories, such as environmental sounds and other common audio domains. This approach yields state-of-the-art performance on a subset of tasks and competitive results across the remainder. Complementary linear probing evaluation mirrors this pattern, yielding clear gains on several benchmarks and strong performance elsewhere. The collective analysis of these results indicates that AaPE serves to mitigate the effects of aliasing without discarding of informative high-frequency content.

Figures (4)

• Figure 1: Overview of the Teacher-student Self-supervised Learning Scheme with AaPE.Aliasing-aware Patch Embedding (AaPE) replaces the ViT patch embedding, performing dynamic subband frequency analysis to extract aliasing-aware features from aliasing-prone bands and fusing them with the standard patch tokens. The class (CLS) token predicts a global summary of the teacher outputs, while a cross-attention predictor performs masked prediction of the teacher’s layer-wise pooled tokens at masked positions. A multi-mask contrastive consistency regularization pulls together masked views of the same input and pushes apart views from different inputs. PosEmb denotes absolute positional embeddingsvit-21.
• Figure 2: Spectra and Gradient Magnitudes for Various Window Functions. The two-sided window sustains usable gradients over wider frequency offsets, while one-sided and Gaussian windows suffer rapid off-target vanishing; this supports our choice of a two-sided exponential window in SBLU for stable subband-wise estimation in aliasing-prone bands. The window parameters $\alpha$ and $\sigma_g$ are normalized so that the peak gradient magnitude equals 1.0.
• Figure 3: Architecture of the Aliasing-aware Patch Embedding (AaPE). AaPE augments the standard ViT patch embedding with aliasing-aware features via three components: (1) Lambda Encoder--takes log-mel spectrogram patches and estimates input-dependent complex kernel parameters $\mathbf{\Lambda}_\text{alias}$ with decay $\alpha$ and frequency $\beta$ per subband using a narrow-and-shallow Transformer and linear projections; (2) Adaptive SBLU--emphasizes aliasing-prone bands with a zero-phase high-pass filter, generates bilateral complex sinusoidal kernels from $\mathbf{\Lambda}_\text{alias}$, and performs memory-efficient adaptive depthwise convolution with grouped pointwise projections to produce an aliasing-focused high-frequency representation; (3) Patch Fusion--independently normalizes and concatenates the standard and aliasing-focused features, then linearly projects to $D$ dimensions to yield the final aliasing-aware patch embedding fed to the ViT encoder.
• Figure 4: Example of Patch-wise Adaptive Estimation of SBLU Kernel Parameters. Top: input spectrogram prior to patchification, which is subsequently patchified and fed into the Lambda Encoder. Middle/Bottom: per-time-patch distributions of the estimated SBLU kernel parameters (decay and frequency) produced by the pre-trained Lambda Encoder, showing adaptive variation in response to the input.