Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gumbel Rao Monte Carlo based Bi-Modal Neural Architecture Search for Audio-Visual Deepfake Detection

Aravinda Reddy PN, Raghavendra Ramachandra, Krothapalli Sreenivasa Rao, Pabitra Mitra Vinod Rathod

TL;DR

Gumbel-Rao Monte Carlo Bi-modal Neural Architecture Search (GRMC-BMNAS), a novel architecture search framework that employs Gumbel-Rao Monte Carlo sampling to optimize multimodal fusion, is proposed.

Abstract

Deepfakes pose a critical threat to biometric authentication systems by generating highly realistic synthetic media. Existing multimodal deepfake detectors often struggle to adapt to diverse data and rely on simple fusion methods. To address these challenges, we propose Gumbel-Rao Monte Carlo Bi-modal Neural Architecture Search (GRMC-BMNAS), a novel architecture search framework that employs Gumbel-Rao Monte Carlo sampling to optimize multimodal fusion. It refines the Straight through Gumbel Softmax (STGS) method by reducing variance with Rao-Blackwellization, stabilizing network training. Using a two-level search approach, the framework optimizes the network architecture, parameters, and performance. Crucial features are efficiently identified from backbone networks, while within the cell structure, a weighted fusion operation integrates information from various sources. By varying parameters such as temperature and number of Monte carlo samples yields an architecture that maximizes classification performance and better generalisation capability. Experimental results on the FakeAVCeleb and SWAN-DF datasets demonstrate an impressive AUC percentage of 95.4\%, achieved with minimal model parameters.

Gumbel Rao Monte Carlo based Bi-Modal Neural Architecture Search for Audio-Visual Deepfake Detection

TL;DR

Gumbel-Rao Monte Carlo Bi-modal Neural Architecture Search (GRMC-BMNAS), a novel architecture search framework that employs Gumbel-Rao Monte Carlo sampling to optimize multimodal fusion, is proposed.

Abstract

Deepfakes pose a critical threat to biometric authentication systems by generating highly realistic synthetic media. Existing multimodal deepfake detectors often struggle to adapt to diverse data and rely on simple fusion methods. To address these challenges, we propose Gumbel-Rao Monte Carlo Bi-modal Neural Architecture Search (GRMC-BMNAS), a novel architecture search framework that employs Gumbel-Rao Monte Carlo sampling to optimize multimodal fusion. It refines the Straight through Gumbel Softmax (STGS) method by reducing variance with Rao-Blackwellization, stabilizing network training. Using a two-level search approach, the framework optimizes the network architecture, parameters, and performance. Crucial features are efficiently identified from backbone networks, while within the cell structure, a weighted fusion operation integrates information from various sources. By varying parameters such as temperature and number of Monte carlo samples yields an architecture that maximizes classification performance and better generalisation capability. Experimental results on the FakeAVCeleb and SWAN-DF datasets demonstrate an impressive AUC percentage of 95.4\%, achieved with minimal model parameters.

Paper Structure

This paper contains 19 sections, 12 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Proposed method: GRMC-BMNAS
  4. Gumbel distribution
  5. Gumbel-max trick
  6. Gumbel-Softmax distribution
  7. Gumbel-Rao Monte Carlo Estimator (GRMC)
  8. Analysing our proposed GRMC-BMNAS
  9. Modality feature extraction
  10. First level search: GRMC relaxation over the cells
  11. Second level: Weighted fusion
  12. Optimizing Neural Architectures through Parameter learning
  13. Experiments and Results
  14. Datasets
  15. Evaluation methodology
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Block diagram showing the proposed GRMC-BMNAS employs a two-stage search to optimize bimodal fusion. The first stage identifies crucial features, while the second stage determines the optimal architecture using a pool of operations.
  • Figure 2: Average entropy plot for two learnable parameters for the proposed GRMC-BMNAS, and the existing STGS-BMNAS pn2024straight and Softmax yin2022bm.
  • Figure 3: Best architecture obtained with K=100 and $\lambda=0.1$