DFALLM: Achieving Generalizable Multitask Deepfake Detection by Optimizing Audio LLM Components

TL;DR

This work tackles the generalization gap in audio deepfake detection by analyzing the encoder–LLM components of Audio Large Language Models (ALLMs). It identifies the audio encoder as the primary bottleneck and demonstrates that acoustically-aware encoders (e.g., Wav2Vec2-BERT) markedly improve generalization, surpassing semantic encoders like Whisper. By pairing a high-capacity encoder with a lightweight yet effective LLM (Qwen2.5-0.5B) and employing multitask prompts, the DFALLM framework achieves state-of-the-art performance on multiple benchmarks and extends to attribution and localization tasks. The results establish the importance of encoder design, frame-rate, and data availability for robust, multitask deepfake detection in real-world, out-of-domain scenarios.

Abstract

Audio deepfake detection has recently garnered public concern due to its implications for security and reliability. Traditional deep learning methods have been widely applied to this task but often lack generalisability when confronted with newly emerging spoofing techniques and more tasks such as spoof attribution recognition rather than simple binary classification. In principle, Large Language Models (LLMs) are considered to possess the needed generalisation capabilities. However, previous research on Audio LLMs (ALLMs) indicates a generalization bottleneck in audio deepfake detection performance, even when sufficient data is available. Consequently, this study investigates the model architecture and examines the effects of the primary components of ALLMs, namely the audio encoder and the text-based LLM. Our experiments demonstrate that the careful selection and combination of audio encoders and text-based LLMs are crucial for unlocking the deepfake detection potential of ALLMs. We further propose an ALLM structure capable of generalizing deepfake detection abilities to out-of-domain spoofing tests and other deepfake tasks, such as spoof positioning and spoof attribution recognition. Our proposed model architecture achieves state-of-the-art (SOTA) performance across multiple datasets, including ASVSpoof2019, InTheWild, and Demopage, with accuracy reaching up to 95.76% on average, and exhibits competitive capabilities in other deepfake detection tasks such as attribution, and localisation compared to SOTA audio understanding models. Data and codes are provided in supplementary materials.

Figures (2)

• Figure 1: DFALLM: The proposed framework takes audio inputs with varying frame rates alongside corresponding textual prompts. The audio signals are first processed by an audio encoder, whose output representations are projected into the text embedding space. These projected embeddings are then combined with the representations obtained from the text tokenizer and jointly processed by a textual LLM to interpret the audio–text input pair and generate task-specific responses. In our setup, the audio encoder is fully trained, while the textual LLM is fine-tuned using the LoRA technique to efficiently adapt its parameters. Both components are modular and can be selected from existing model libraries to enable systematic investigation of their respective contributions.
• Figure 2: Performance on Qwen2.5-0.5B with Wav2Vec-Bert across datasets with different sample size for each component source dataset