Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A benchmark of state-of-the-art sound event detection systems evaluated on synthetic soundscapes

Francesca Ronchini, Romain Serizel

TL;DR

The paper benchmarks state-of-the-art SED submissions from DCASE 2021 Task 4 using two PSDS-based evaluation scenarios $PSDS_1$ and $PSDS_2$ to explore how architectural choices aimed at fine vs coarse temporal segmentation affect robustness to varying TNTSNR and non-target events. It relies on the DESED synthetic soundscapes and a suite of synthetic evaluation sets that manipulate target-to-nontarget SNR, onset time, and non-target presence. The results show that systems optimized for coarse segmentation are more robust to TNTSNR and time localization due to augmentation and classification strengths, while fine-segmentation systems tend to produce more short false positives when non-targets are present. These findings inform the design and evaluation of SED systems under synthetic benchmarks and have implications for deployment in real-world noisy environments.

Abstract

This paper proposes a benchmark of submissions to Detection and Classification Acoustic Scene and Events 2021 Challenge (DCASE) Task 4 representing a sampling of the state-of-the-art in Sound Event Detection task. The submissions are evaluated according to the two polyphonic sound detection score scenarios proposed for the DCASE 2021 Challenge Task 4, which allow to make an analysis on whether submissions are designed to perform fine-grained temporal segmentation, coarse-grained temporal segmentation, or have been designed to be polyvalent on the scenarios proposed. We study the solutions proposed by participants to analyze their robustness to varying level target to non-target signal-to-noise ratio and to temporal localization of target sound events. A last experiment is proposed in order to study the impact of non-target events on systems outputs. Results show that systems adapted to provide coarse segmentation outputs are more robust to different target to non-target signal-to-noise ratio and, with the help of specific data augmentation methods, they are more robust to time localization of the original event. Results of the last experiment display that systems tend to spuriously predict short events when non-target events are present. This is particularly true for systems that are tailored to have a fine segmentation.

A benchmark of state-of-the-art sound event detection systems evaluated on synthetic soundscapes

TL;DR

The paper benchmarks state-of-the-art SED submissions from DCASE 2021 Task 4 using two PSDS-based evaluation scenarios and to explore how architectural choices aimed at fine vs coarse temporal segmentation affect robustness to varying TNTSNR and non-target events. It relies on the DESED synthetic soundscapes and a suite of synthetic evaluation sets that manipulate target-to-nontarget SNR, onset time, and non-target presence. The results show that systems optimized for coarse segmentation are more robust to TNTSNR and time localization due to augmentation and classification strengths, while fine-segmentation systems tend to produce more short false positives when non-targets are present. These findings inform the design and evaluation of SED systems under synthetic benchmarks and have implications for deployment in real-world noisy environments.

Abstract

This paper proposes a benchmark of submissions to Detection and Classification Acoustic Scene and Events 2021 Challenge (DCASE) Task 4 representing a sampling of the state-of-the-art in Sound Event Detection task. The submissions are evaluated according to the two polyphonic sound detection score scenarios proposed for the DCASE 2021 Challenge Task 4, which allow to make an analysis on whether submissions are designed to perform fine-grained temporal segmentation, coarse-grained temporal segmentation, or have been designed to be polyvalent on the scenarios proposed. We study the solutions proposed by participants to analyze their robustness to varying level target to non-target signal-to-noise ratio and to temporal localization of target sound events. A last experiment is proposed in order to study the impact of non-target events on systems outputs. Results show that systems adapted to provide coarse segmentation outputs are more robust to different target to non-target signal-to-noise ratio and, with the help of specific data augmentation methods, they are more robust to time localization of the original event. Results of the last experiment display that systems tend to spuriously predict short events when non-target events are present. This is particularly true for systems that are tailored to have a fine segmentation.
Paper Structure (15 sections, 1 equation, 4 figures, 3 tables)

This paper contains 15 sections, 1 equation, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Datasets and task setup
  3. Task setup and evaluation metrics
  4. Datasets
  5. DESED dataset
  6. Synthetic evaluation datasets
  7. Reference synthetic soundscapes evaluation set
  8. Synthetic set with varying TNTSNR
  9. Synthetic set with varying onset time
  10. Impact of the evaluation scenario
  11. ANALYSIS ON THE SYNTHETIC DATASETS
  12. Impact of TNTSNR
  13. Impact of time localization of the original event
  14. Impact of non-target sound events
  15. CONCLUSIONS

Figures (4)

  • Figure 1: PSDS_1 results for systems selected for scenario 1, evaluated with reference evaluation set and sets with varying TNTSNR.
  • Figure 2: PSDS_2 results for systems selected for scenario 1, evaluated with reference evaluation set and sets with varying TNTSNR.
  • Figure 3: PSDS_2 results for systems selected for scenario 2, evaluated with reference evaluation set and sets with varying TNTSNR.
  • Figure 4: PSDS_1 results for systems selected for scenario 1, evaluated with synthetic set with varying onset time.