Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Differences between Human Perception and Model Inference in Audio Event Recognition

Yizhou Tan, Yanru Wu, Yuanbo Hou, Xin Xu, Hui Bu, Shengchen Li, Dick Botteldooren, Mark D. Plumbley

TL;DR

This paper constructs a Multi-Annotated Foreground Audio Event Recognition (MAFAR) dataset, which comprises audio recordings labelled by 10 professional annotators that facilitates the quantification of semantic importance and analysis of human perception.

Abstract

Audio Event Recognition (AER) traditionally focuses on detecting and identifying audio events. Most existing AER models tend to detect all potential events without considering their varying significance across different contexts. This makes the AER results detected by existing models often have a large discrepancy with human auditory perception. Although this is a critical and significant issue, it has not been extensively studied by the Detection and Classification of Sound Scenes and Events (DCASE) community because solving it is time-consuming and labour-intensive. To address this issue, this paper introduces the concept of semantic importance in AER, focusing on exploring the differences between human perception and model inference. This paper constructs a Multi-Annotated Foreground Audio Event Recognition (MAFAR) dataset, which comprises audio recordings labelled by 10 professional annotators. Through labelling frequency and variance, the MAFAR dataset facilitates the quantification of semantic importance and analysis of human perception. By comparing human annotations with the predictions of ensemble pre-trained models, this paper uncovers a significant gap between human perception and model inference in both semantic identification and existence detection of audio events. Experimental results reveal that human perception tends to ignore subtle or trivial events in the event semantic identification, while model inference is easily affected by events with noises. Meanwhile, in event existence detection, models are usually more sensitive than humans.

Exploring Differences between Human Perception and Model Inference in Audio Event Recognition

TL;DR

This paper constructs a Multi-Annotated Foreground Audio Event Recognition (MAFAR) dataset, which comprises audio recordings labelled by 10 professional annotators that facilitates the quantification of semantic importance and analysis of human perception.

Abstract

Audio Event Recognition (AER) traditionally focuses on detecting and identifying audio events. Most existing AER models tend to detect all potential events without considering their varying significance across different contexts. This makes the AER results detected by existing models often have a large discrepancy with human auditory perception. Although this is a critical and significant issue, it has not been extensively studied by the Detection and Classification of Sound Scenes and Events (DCASE) community because solving it is time-consuming and labour-intensive. To address this issue, this paper introduces the concept of semantic importance in AER, focusing on exploring the differences between human perception and model inference. This paper constructs a Multi-Annotated Foreground Audio Event Recognition (MAFAR) dataset, which comprises audio recordings labelled by 10 professional annotators. Through labelling frequency and variance, the MAFAR dataset facilitates the quantification of semantic importance and analysis of human perception. By comparing human annotations with the predictions of ensemble pre-trained models, this paper uncovers a significant gap between human perception and model inference in both semantic identification and existence detection of audio events. Experimental results reveal that human perception tends to ignore subtle or trivial events in the event semantic identification, while model inference is easily affected by events with noises. Meanwhile, in event existence detection, models are usually more sensitive than humans.
Paper Structure (9 sections, 4 equations, 2 figures)

This paper contains 9 sections, 4 equations, 2 figures.

Table of Contents

  1. Introduction
  2. MAFAR Dataset
  3. Annotation Guidelines for Humans
  4. Label Alignment via Large Language Model
  5. Benchmark Methods and Resutls
  6. Difference in Event Semantic Identification
  7. Difference in Event Existence Detection
  8. Discussion
  9. Conclusion

Figures (2)

  • Figure 1: The process of label alignment is assisted by GPT-4.
  • Figure 4: The ensemble results of the existing AudioSet pre-trained models in event existence detection. The ground truth threshold of the existence of interested events is based on the total amount of labelled events from 10 annotators. Since there could be more than one event in segments, the ground truth threshold could exceed 10.