Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detection of Deepfake Environmental Audio

Hafsa Ouajdi, Oussama Hadder, Modan Tailleur, Mathieu Lagrange, Laurie M. Heller

TL;DR

This work addresses the understudied problem of detecting deepfake environmental audio by proposing a CLAP-based detector evaluated on the DCASE 2023 Foley dataset. It leverages pre-trained audio embeddings (CLAP, VGGish, PANNs) fed through a lightweight MLP to perform binary classification, achieving an average detection accuracy of about 98% across 44 synthesizers, with CLAP-based embeddings outperforming VGGish by roughly 10%. The study also includes a human listening test that reveals perceptual cues and limitations of both machines and generators, suggesting directions for improvement in future synthesis systems. The findings indicate that practical and accurate detection of fake environmental sounds is feasible today, with implications for authentication and safety in real-world audio workflows.

Abstract

With the ever-rising quality of deep generative models, it is increasingly important to be able to discern whether the audio data at hand have been recorded or synthesized. Although the detection of fake speech signals has been studied extensively, this is not the case for the detection of fake environmental audio. We propose a simple and efficient pipeline for detecting fake environmental sounds based on the CLAP audio embedding. We evaluate this detector using audio data from the 2023 DCASE challenge task on Foley sound synthesis. Our experiments show that fake sounds generated by 44 state-of-the-art synthesizers can be detected on average with 98% accuracy. We show that using an audio embedding learned on environmental audio is beneficial over a standard VGGish one as it provides a 10% increase in detection performance. Informal listening to Incorrect Negative examples demonstrates audible features of fake sounds missed by the detector such as distortion and implausible background noise.

Detection of Deepfake Environmental Audio

TL;DR

This work addresses the understudied problem of detecting deepfake environmental audio by proposing a CLAP-based detector evaluated on the DCASE 2023 Foley dataset. It leverages pre-trained audio embeddings (CLAP, VGGish, PANNs) fed through a lightweight MLP to perform binary classification, achieving an average detection accuracy of about 98% across 44 synthesizers, with CLAP-based embeddings outperforming VGGish by roughly 10%. The study also includes a human listening test that reveals perceptual cues and limitations of both machines and generators, suggesting directions for improvement in future synthesis systems. The findings indicate that practical and accurate detection of fake environmental sounds is feasible today, with implications for authentication and safety in real-world audio workflows.

Abstract

With the ever-rising quality of deep generative models, it is increasingly important to be able to discern whether the audio data at hand have been recorded or synthesized. Although the detection of fake speech signals has been studied extensively, this is not the case for the detection of fake environmental audio. We propose a simple and efficient pipeline for detecting fake environmental sounds based on the CLAP audio embedding. We evaluate this detector using audio data from the 2023 DCASE challenge task on Foley sound synthesis. Our experiments show that fake sounds generated by 44 state-of-the-art synthesizers can be detected on average with 98% accuracy. We show that using an audio embedding learned on environmental audio is beneficial over a standard VGGish one as it provides a 10% increase in detection performance. Informal listening to Incorrect Negative examples demonstrates audible features of fake sounds missed by the detector such as distortion and implausible background noise.
Paper Structure (18 sections, 1 equation, 1 figure, 3 tables)

This paper contains 18 sections, 1 equation, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed approach
  4. Experiments
  5. Dataset
  6. Training Procedure
  7. Results
  8. Inference Time
  9. Overall Accuracy
  10. Statistical Analysis
  11. Class-Wise Accuracy
  12. Accuracy vs. Generator Quality
  13. Listening Test
  14. Data
  15. Methods
  16. ...and 3 more sections

Figures (1)

  • Figure 1: Overview of the pipeline used in the experiments for the Deepfake detection, with a representation of the MLP's network architecture. The value of dim depends on the embedding method used.