Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Language Model Capabilities for Sound Event Detection

Hualei Wang, Jianguo Mao, Zhifang Guo, Jiarui Wan, Hong Liu, Xiangdong Wang

TL;DR

This work tackles the challenge of precise sound event detection by leveraging language models to describe events and their temporal locations, bridging audio and text modalities. It introduces SED-LM, an end-to-end framework where a pretrained audio encoder and a language-model-based decoder are connected through cross-attention, with autoregressive text generation guiding event labeling. Experiments on the DCASE 2023 Task4 dataset show that BEATs encoder with a BERT decoder achieves strong EB-F1 and IB-F1 scores, and that performance is heavily influenced by text templates, decoding strategy, and text augmentation. The approach demonstrates the viability of multimodal, language-guided SED for improved timestamp precision and event classification, offering a flexible framework adaptable to different audio features and language models.

Abstract

Large language models reveal deep comprehension and fluent generation in the field of multi-modality. Although significant advancements have been achieved in audio multi-modality, existing methods are rarely leverage language model for sound event detection (SED). In this work, we propose an end-to-end framework for understanding audio features while simultaneously generating sound event and their temporal location. Specifically, we employ pretrained acoustic models to capture discriminative features across different categories and language models for autoregressive text generation. Conventional methods generally struggle to obtain features in pure audio domain for classification. In contrast, our framework utilizes the language model to flexibly understand abundant semantic context aligned with the acoustic representation. The experimental results showcase the effectiveness of proposed method in enhancing timestamps precision and event classification.

Leveraging Language Model Capabilities for Sound Event Detection

TL;DR

This work tackles the challenge of precise sound event detection by leveraging language models to describe events and their temporal locations, bridging audio and text modalities. It introduces SED-LM, an end-to-end framework where a pretrained audio encoder and a language-model-based decoder are connected through cross-attention, with autoregressive text generation guiding event labeling. Experiments on the DCASE 2023 Task4 dataset show that BEATs encoder with a BERT decoder achieves strong EB-F1 and IB-F1 scores, and that performance is heavily influenced by text templates, decoding strategy, and text augmentation. The approach demonstrates the viability of multimodal, language-guided SED for improved timestamp precision and event classification, offering a flexible framework adaptable to different audio features and language models.

Abstract

Large language models reveal deep comprehension and fluent generation in the field of multi-modality. Although significant advancements have been achieved in audio multi-modality, existing methods are rarely leverage language model for sound event detection (SED). In this work, we propose an end-to-end framework for understanding audio features while simultaneously generating sound event and their temporal location. Specifically, we employ pretrained acoustic models to capture discriminative features across different categories and language models for autoregressive text generation. Conventional methods generally struggle to obtain features in pure audio domain for classification. In contrast, our framework utilizes the language model to flexibly understand abundant semantic context aligned with the acoustic representation. The experimental results showcase the effectiveness of proposed method in enhancing timestamps precision and event classification.
Paper Structure (20 sections, 4 equations, 3 figures, 5 tables)

This paper contains 20 sections, 4 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Overview of proposed SED-LM model
  4. Generative SED
  5. Audio extraction: from spectrograms to features
  6. Language models: from audio tokens to generated text
  7. Cross attention: integrate audio and text
  8. Text augmentation
  9. Experiments Setup
  10. Datasets
  11. Experimental Settings
  12. Evaluation metrics
  13. Experiments Result
  14. Comparison between proposed and other method for SED
  15. Integrated feature analysing
  16. ...and 5 more sections

Figures (3)

  • Figure 1: We represent the DCASE2023 baseline SED systems and language model for automatic SED. (a) shows baseline SED system. (b) is the architecture of SED-LM. The module in the dashed block is a transformer layer with cross-attention.
  • Figure 2: The trained HT-SAT and BEATs feature
  • Figure 3: Y0s6Vt4i3iiQ_40.000_50.000.wav's similarity. The electric toothbrush event's similarity between frame and corresponding text is represented by red color. The speech is represented by blue color.