Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MAVOS-DD: Multilingual Audio-Video Open-Set Deepfake Detection Benchmark

Florinel-Alin Croitoru, Vlad Hondru, Marius Popescu, Radu Tudor Ionescu, Fahad Shahbaz Khan, Mubarak Shah

TL;DR

MAVOS-DD tackles the critical problem of robust multilingual audio-video deepfake detection by introducing a large-scale open-set benchmark that spans eight languages and seven deepfake generation methods. The authors evaluate state-of-the-art detectors under four evaluation modes—in-domain and three open-set setups—revealing substantial generalization gaps when models face unseen generation techniques and languages. The dataset design deliberately excludes certain languages and models during training to simulate realistic open-world conditions, and results show multimodal detectors outperform unimodal baselines but still struggle with open-set generalization. This work provides a substantial resource for developing more robust, cross-language deepfake detectors and highlights the need for open-set-focused approaches and continual dataset updates to match the pace of generative advances.

Abstract

We present the first large-scale open-set benchmark for multilingual audio-video deepfake detection. Our dataset comprises over 250 hours of real and fake videos across eight languages, with 60% of data being generated. For each language, the fake videos are generated with seven distinct deepfake generation models, selected based on the quality of the generated content. We organize the training, validation and test splits such that only a subset of the chosen generative models and languages are available during training, thus creating several challenging open-set evaluation setups. We perform experiments with various pre-trained and fine-tuned deepfake detectors proposed in recent literature. Our results show that state-of-the-art detectors are not currently able to maintain their performance levels when tested in our open-set scenarios. We publicly release our data and code at: https://huggingface.co/datasets/unibuc-cs/MAVOS-DD.

MAVOS-DD: Multilingual Audio-Video Open-Set Deepfake Detection Benchmark

TL;DR

MAVOS-DD tackles the critical problem of robust multilingual audio-video deepfake detection by introducing a large-scale open-set benchmark that spans eight languages and seven deepfake generation methods. The authors evaluate state-of-the-art detectors under four evaluation modes—in-domain and three open-set setups—revealing substantial generalization gaps when models face unseen generation techniques and languages. The dataset design deliberately excludes certain languages and models during training to simulate realistic open-world conditions, and results show multimodal detectors outperform unimodal baselines but still struggle with open-set generalization. This work provides a substantial resource for developing more robust, cross-language deepfake detectors and highlights the need for open-set-focused approaches and continual dataset updates to match the pace of generative advances.

Abstract

We present the first large-scale open-set benchmark for multilingual audio-video deepfake detection. Our dataset comprises over 250 hours of real and fake videos across eight languages, with 60% of data being generated. For each language, the fake videos are generated with seven distinct deepfake generation models, selected based on the quality of the generated content. We organize the training, validation and test splits such that only a subset of the chosen generative models and languages are available during training, thus creating several challenging open-set evaluation setups. We perform experiments with various pre-trained and fine-tuned deepfake detectors proposed in recent literature. Our results show that state-of-the-art detectors are not currently able to maintain their performance levels when tested in our open-set scenarios. We publicly release our data and code at: https://huggingface.co/datasets/unibuc-cs/MAVOS-DD.
Paper Structure (7 sections, 5 figures, 3 tables)

This paper contains 7 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset
  4. Experiments
  5. Broader Impact and Limitations
  6. Conclusion and Future Work
  7. Ethical Statement

Figures (5)

  • Figure 1: In MAVOS-DD, the training set and in-domain test set contain real and fake videos sampled from the same distribution, comprising six languages and four generative models. The open-set model test set extends the in-domain test set with fake samples generated by unseen models (Sonic, HifiFace, Roop). The open-set language test set extends the in-domain test set with samples in unseen languages (German and Hindi). The open-set full test set adds samples generated by unseen models in unseen languages. One fake sample from each data distribution is shown on the right-hand side. Best viewed in color.
  • Figure 2: Distribution of videos per language and per generative method. MAVOS-DD comprises videos in eight languages, generated with seven methods. The languages are coded as follows: Arabic (AR), English (EN), German (DE), Hindi (HI), Mandarin (MD), Romanian (RO), Russian (RU) and Spanish (ES).
  • Figure 3: Fake video frames generated by each of the seven deepfake methods. Best viewed in color.
  • Figure 4: Confusion matrices obtained by AVFF, MRDF and TALL after fine-tuning them on MAVOS-DD.
  • Figure 5: A real video and its corresponding fake sample generated using LivePortrait. The MRDF detector incorrectly classifies the fake sample as real. Best viewed in color.