Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deformable Audio Transformer for Audio Event Detection

Wentao Zhu

TL;DR

This paper addresses the high computational cost of self-attention in Transformer-based audio models by introducing a deformable audio Transformer (DATAR) with a pyramid backbone and a learnable audio offset generator (AOG) that focuses attention on informative spectrogram patches. It further improves attention reliability with a learnable input adaptor that injects trainable signals to better inform the deformable sampling. Empirical results on Kinetics-Sounds, Epic-Kitchens-100, and VGGSound show large improvements in top-1 accuracy over prior state-of-the-art methods, with reduced compute (MACs) compared to baselines. The approach enables efficient, data-adaptive audio event detection and offers practical benefits for real-world, resource-constrained deployments.

Abstract

Transformers have achieved promising results on a variety of tasks. However, the quadratic complexity in self-attention computation has limited the applications, especially in low-resource settings and mobile or edge devices. Existing works have proposed to exploit hand-crafted attention patterns to reduce computation complexity. However, such hand-crafted patterns are data-agnostic and may not be optimal. Hence, it is likely that relevant keys or values are being reduced, while less important ones are still preserved. Based on this key insight, we propose a novel deformable audio Transformer for audio recognition, named DATAR, where a deformable attention equipping with a pyramid transformer backbone is constructed and learnable. Such an architecture has been proven effective in prediction tasks,~\textit{e.g.}, event classification. Moreover, we identify that the deformable attention map computation may over-simplify the input feature, which can be further enhanced. Hence, we introduce a learnable input adaptor to alleviate this issue, and DATAR achieves state-of-the-art performance.

Deformable Audio Transformer for Audio Event Detection

TL;DR

This paper addresses the high computational cost of self-attention in Transformer-based audio models by introducing a deformable audio Transformer (DATAR) with a pyramid backbone and a learnable audio offset generator (AOG) that focuses attention on informative spectrogram patches. It further improves attention reliability with a learnable input adaptor that injects trainable signals to better inform the deformable sampling. Empirical results on Kinetics-Sounds, Epic-Kitchens-100, and VGGSound show large improvements in top-1 accuracy over prior state-of-the-art methods, with reduced compute (MACs) compared to baselines. The approach enables efficient, data-adaptive audio event detection and offers practical benefits for real-world, resource-constrained deployments.

Abstract

Transformers have achieved promising results on a variety of tasks. However, the quadratic complexity in self-attention computation has limited the applications, especially in low-resource settings and mobile or edge devices. Existing works have proposed to exploit hand-crafted attention patterns to reduce computation complexity. However, such hand-crafted patterns are data-agnostic and may not be optimal. Hence, it is likely that relevant keys or values are being reduced, while less important ones are still preserved. Based on this key insight, we propose a novel deformable audio Transformer for audio recognition, named DATAR, where a deformable attention equipping with a pyramid transformer backbone is constructed and learnable. Such an architecture has been proven effective in prediction tasks,~\textit{e.g.}, event classification. Moreover, we identify that the deformable attention map computation may over-simplify the input feature, which can be further enhanced. Hence, we introduce a learnable input adaptor to alleviate this issue, and DATAR achieves state-of-the-art performance.
Paper Structure (7 sections, 6 equations, 3 figures, 4 tables)

This paper contains 7 sections, 6 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. Deformable Audio Transformer
  5. Learnable Input Adaptor
  6. Experiments
  7. Conclusion

Figures (3)

  • Figure 1: Visualization of the effectiveness of proposed learnable input adaptor, which enhances relatively important parts of the original signal, a log-compressed mel-scaled spectrogram, to become more distinguishable, colored by green dash ovals.
  • Figure 2: Illustration of deformable audio Transformer.
  • Figure 3: Illustration of spectrogram deforming processing. Audio offset generator in $\S3.1$ calculates offsets used in deformable attention. When applying the deformable attention on the audio spectrum, for each query token, we select the most informative spectrum patch to conduct the self-attention. Based on the query location, DATAR learns the offset to localize the target spectrum patches.