Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AMAuT: A Flexible and Efficient Multiview Audio Transformer Framework Trained from Scratch

Weichuang Shao, Iman Yi Liao, Tomas Henrique Bode Maul, Tissa Chandesa

TL;DR

AMAuT tackles the limitations of fixed input rates and durations in pretrained audio transformers by training from scratch and supporting arbitrary sample rates and lengths. It introduces augmentation-driven multiview learning, a conv1 + conv7 + conv1 1D CNN bottleneck, vertical embedding with CLS+TAL tokens, and test-time adaptation/augmentation ($TTA^2$) to preserve performance. On five benchmarks (AudioMNIST, SpeechCommands V1/V2, VocalSound, CochlScene), it matches or exceeds several pre-trained baselines while using under 3% of their GPU hours, demonstrating substantial efficiency gains. The approach enables practical deployment of state-of-the-art audio classification in computationally constrained settings, including embedded and real-time environments, by removing reliance on large-scale pre-training and fixed-input configurations.

Abstract

Recent foundational models, SSAST, EAT, HuBERT, Qwen-Audio, and Audio Flamingo, achieve top-tier results across standard audio benchmarks but are limited by fixed input rates and durations, hindering their reusability. This paper introduces the Augmentation-driven Multiview Audio Transformer (AMAuT), a training-from-scratch framework that eliminates the dependency on pre-trained weights while supporting arbitrary sample rates and audio lengths. AMAuT integrates four key components: (1) augmentation-driven multiview learning for robustness, (2) a conv1 + conv7 + conv1 one-dimensional CNN bottleneck for stable temporal encoding, (3) dual CLS + TAL tokens for bidirectional context representation, and (4) test-time adaptation/augmentation (TTA^2) to improve inference reliability. Experiments on five public benchmarks, AudioMNIST, SpeechCommands V1 & V2, VocalSound, and CochlScene, show that AMAuT achieves accuracies up to 99.8% while consuming less than 3% of the GPU hours required by comparable pre-trained models. Thus, AMAuT presents a highly efficient and flexible alternative to large pre-trained models, making state-of-the-art audio classification accessible in computationally constrained settings.

AMAuT: A Flexible and Efficient Multiview Audio Transformer Framework Trained from Scratch

TL;DR

AMAuT tackles the limitations of fixed input rates and durations in pretrained audio transformers by training from scratch and supporting arbitrary sample rates and lengths. It introduces augmentation-driven multiview learning, a conv1 + conv7 + conv1 1D CNN bottleneck, vertical embedding with CLS+TAL tokens, and test-time adaptation/augmentation () to preserve performance. On five benchmarks (AudioMNIST, SpeechCommands V1/V2, VocalSound, CochlScene), it matches or exceeds several pre-trained baselines while using under 3% of their GPU hours, demonstrating substantial efficiency gains. The approach enables practical deployment of state-of-the-art audio classification in computationally constrained settings, including embedded and real-time environments, by removing reliance on large-scale pre-training and fixed-input configurations.

Abstract

Recent foundational models, SSAST, EAT, HuBERT, Qwen-Audio, and Audio Flamingo, achieve top-tier results across standard audio benchmarks but are limited by fixed input rates and durations, hindering their reusability. This paper introduces the Augmentation-driven Multiview Audio Transformer (AMAuT), a training-from-scratch framework that eliminates the dependency on pre-trained weights while supporting arbitrary sample rates and audio lengths. AMAuT integrates four key components: (1) augmentation-driven multiview learning for robustness, (2) a conv1 + conv7 + conv1 one-dimensional CNN bottleneck for stable temporal encoding, (3) dual CLS + TAL tokens for bidirectional context representation, and (4) test-time adaptation/augmentation (TTA^2) to improve inference reliability. Experiments on five public benchmarks, AudioMNIST, SpeechCommands V1 & V2, VocalSound, and CochlScene, show that AMAuT achieves accuracies up to 99.8% while consuming less than 3% of the GPU hours required by comparable pre-trained models. Thus, AMAuT presents a highly efficient and flexible alternative to large pre-trained models, making state-of-the-art audio classification accessible in computationally constrained settings.
Paper Structure (28 sections, 11 equations, 4 figures, 12 tables, 1 algorithm)

This paper contains 28 sections, 11 equations, 4 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Transformer-based Audio Models
  4. Test-Time Domain Adaptation
  5. Methodology
  6. Overview
  7. Multiview Augmentation
  8. Spectrogram and Horizontal Tokenization
  9. 1D CNNs
  10. Vertical Embedding
  11. Classifier
  12. Test-Time Adaptation / Augmentation ($TTA^2$)
  13. Learning Rate and Optimization
  14. Experiments and Results
  15. Datasets
  16. ...and 13 more sections

Figures (4)

  • Figure 1: Overall architecture of the proposed AMAuT framework. Here, C denotes the number of classes, while N, F, T, and K depend on the dataset.
  • Figure 2: Structure of the 1D CNN module used in AMAuT
  • Figure 3: Architecture of the Vertical Embedding module in AMAuT
  • Figure 4: Classifier architectures used in AMAuT for different audio durations. (a) For audio $>$1s, only CLS + TAL tokens are used, followed by a 1D CNN and a fully connected (FC) layer. (b) For 1s inputs, using all features with AvgPool and FC layers.