ElasticAST: An Audio Spectrogram Transformer for All Length and Resolutions

TL;DR

ElasticAST addresses the fixed-length input constraint of Audio Spectrogram Transformers by enabling variable-length and variable-resolution audio processing during both training and inference. It achieves this with a simple extension: sequence packing, Mask Self-Attention to keep attention within each sample, and Mask Attention Pooling to generate per-sample representations, all with minimal changes and a per-row token limit $L'$ (default 2048). Experiments across AudioSet, VGGSound, VoxCeleb, and Epic-Sounds show ElasticAST maintains or improves performance across lengths and resolutions, reduces reliance on trimming/padding, and adapts to different computational budgets. The approach yields a single, efficient model that is particularly advantageous for in-the-wild and multimodal scenarios where audio lengths vary naturally, enabling robust alignment, retrieval, and generation tasks without retraining for each length or resolution.

Abstract

Transformers have rapidly overtaken CNN-based architectures as the new standard in audio classification. Transformer-based models, such as the Audio Spectrogram Transformers (AST), also inherit the fixed-size input paradigm from CNNs. However, this leads to performance degradation for ASTs in the inference when input lengths vary from the training. This paper introduces an approach that enables the use of variable-length audio inputs with AST models during both training and inference. By employing sequence packing, our method ElasticAST, accommodates any audio length during training, thereby offering flexibility across all lengths and resolutions at the inference. This flexibility allows ElasticAST to maintain evaluation capabilities at various lengths or resolutions and achieve similar performance to standard ASTs trained at specific lengths or resolutions. Moreover, experiments demonstrate ElasticAST's better performance when trained and evaluated on native-length audio datasets.

Figures (5)

• Figure 1: Standard ASTs vs. ElasticAST. Standard ASTs' performance degrades when evaluated on audio lengths different from their trained lengths, while ElasticAST remains flexible to varying lengths.
• Figure 2: Variable Length Datasets and standard AST training input. Due to the fixed-length processing constraints, ASTs discard informative tokens and introduce non-informative tokens.
• Figure 3: Our ElasticAST framework.
• Figure 4: Results on variable native length audios.
• Figure 5: Results on various resolutions.