Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Investigating the Viability of Employing Multi-modal Large Language Models in the Context of Audio Deepfake Detection

Akanksha Chuchra, Shukesh Reddy, Sudeepta Mishra, Abhijit Das, Abhinav Dhall

TL;DR

This work investigates whether audio deepfake detection can benefit from Multimodal Large Language Models by reframing the task as Audio Question-Answering and evaluating two contemporary MLLMs (Qwen2-Audio-7B-Instruct and SALMONN) in zero-shot and LoRA-based finetuning regimes. It demonstrates that zero-shot performance is near random, but modest, task-specific fine-tuning with carefully designed prompts yields strong in-domain results while generalizing less effectively to out-of-domain data. The study highlights the potential of instruction-guided audio reasoning while underscoring the need for domain-specific data and architectures to achieve robust cross-domain generalization. Overall, it provides a foundational evaluation of MLLMs for audio deepfake detection and outlines concrete directions for improving generalization and explainability in audio-forensics tasks.

Abstract

While Vision-Language Models (VLMs) and Multimodal Large Language Models (MLLMs) have shown strong generalisation in detecting image and video deepfakes, their use for audio deepfake detection remains largely unexplored. In this work, we aim to explore the potential of MLLMs for audio deepfake detection. Combining audio inputs with a range of text prompts as queries to find out the viability of MLLMs to learn robust representations across modalities for audio deepfake detection. Therefore, we attempt to explore text-aware and context-rich, question-answer based prompts with binary decisions. We hypothesise that such a feature-guided reasoning will help in facilitating deeper multimodal understanding and enable robust feature learning for audio deepfake detection. We evaluate the performance of two MLLMs, Qwen2-Audio-7B-Instruct and SALMONN, in two evaluation modes: (a) zero-shot and (b) fine-tuned. Our experiments demonstrate that combining audio with a multi-prompt approach could be a viable way forward for audio deepfake detection. Our experiments show that the models perform poorly without task-specific training and struggle to generalise to out-of-domain data. However, they achieve good performance on in-domain data with minimal supervision, indicating promising potential for audio deepfake detection.

Investigating the Viability of Employing Multi-modal Large Language Models in the Context of Audio Deepfake Detection

TL;DR

This work investigates whether audio deepfake detection can benefit from Multimodal Large Language Models by reframing the task as Audio Question-Answering and evaluating two contemporary MLLMs (Qwen2-Audio-7B-Instruct and SALMONN) in zero-shot and LoRA-based finetuning regimes. It demonstrates that zero-shot performance is near random, but modest, task-specific fine-tuning with carefully designed prompts yields strong in-domain results while generalizing less effectively to out-of-domain data. The study highlights the potential of instruction-guided audio reasoning while underscoring the need for domain-specific data and architectures to achieve robust cross-domain generalization. Overall, it provides a foundational evaluation of MLLMs for audio deepfake detection and outlines concrete directions for improving generalization and explainability in audio-forensics tasks.

Abstract

While Vision-Language Models (VLMs) and Multimodal Large Language Models (MLLMs) have shown strong generalisation in detecting image and video deepfakes, their use for audio deepfake detection remains largely unexplored. In this work, we aim to explore the potential of MLLMs for audio deepfake detection. Combining audio inputs with a range of text prompts as queries to find out the viability of MLLMs to learn robust representations across modalities for audio deepfake detection. Therefore, we attempt to explore text-aware and context-rich, question-answer based prompts with binary decisions. We hypothesise that such a feature-guided reasoning will help in facilitating deeper multimodal understanding and enable robust feature learning for audio deepfake detection. We evaluate the performance of two MLLMs, Qwen2-Audio-7B-Instruct and SALMONN, in two evaluation modes: (a) zero-shot and (b) fine-tuned. Our experiments demonstrate that combining audio with a multi-prompt approach could be a viable way forward for audio deepfake detection. Our experiments show that the models perform poorly without task-specific training and struggle to generalise to out-of-domain data. However, they achieve good performance on in-domain data with minimal supervision, indicating promising potential for audio deepfake detection.
Paper Structure (14 sections, 5 equations, 2 figures, 5 tables)

This paper contains 14 sections, 5 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Proposed Methodology
  4. Problem Formulation
  5. Text Prompts
  6. Zero-Shot Evaluation
  7. Fine-tuning the MLLMs
  8. Experiments & Results
  9. Dataset
  10. Experimental Setup
  11. Implementation Details & Evaluation Metrics
  12. Results
  13. Discussion and Challenges
  14. Conclusion & Future Work

Figures (2)

  • Figure 1: Left: Audio MLLM-based approach for audio deepfake detection, formulated as an Audio Question-Answering (AQA) task. Right: Traditional approaches, which rely on either end-to-end architectures or pretrained model (PTM) feature extractors followed by a classifier for discrete label prediction.
  • Figure 2: Overview of Proposed Audio Deepfake Detection using fine tuning of MLLMs. Fine-tuning is performed using Low-Rank Adaptation (LoRA), enabling efficient adaptation with minimal trainable parameters.