Towards Explicit Acoustic Evidence Perception in Audio LLMs for Speech Deepfake Detection

TL;DR

This paper tackles the vulnerability of audio LLM-based speech deepfake detectors to semantic-dominant reasoning that can overlook fine-grained acoustic artifacts. It introduces SDD-APALLM, which explicitly exposes time-frequency acoustic evidence through Constant-Q Transform–based visuals alongside raw audio, enabling joint reasoning in an audio LLM without altering the pretrained encoder. The approach preserves semantic understanding while grounding decisions in acoustically grounded cues, improving robustness under domain shifts and cross-lingual scenarios. Empirical results on ASVspoof benchmarks, along with interpretability analyses, show consistent gains over audio-only supervision and existing methods, with evidence tokens actively contributing to inference.

Abstract

Speech deepfake detection (SDD) focuses on identifying whether a given speech signal is genuine or has been synthetically generated. Existing audio large language model (LLM)-based methods excel in content understanding; however, their predictions are often biased toward semantically correlated cues, which results in fine-grained acoustic artifacts being overlooked during the decisionmaking process. Consequently, fake speech with natural semantics can bypass detectors despite harboring subtle acoustic anomalies; this suggests that the challenge stems not from the absence of acoustic data, but from its inadequate accessibility when semantic-dominant reasoning prevails. To address this issue, we investigate SDD within the audio LLM paradigm and introduce SDD with Auditory Perception-enhanced Audio Large Language Model (SDD-APALLM), an acoustically enhanced framework designed to explicitly expose fine-grained time-frequency evidence as accessible acoustic cues. By combining raw audio with structured spectrograms, the proposed framework empowers audio LLMs to more effectively capture subtle acoustic inconsistencies without compromising their semantic understanding. Experimental results indicate consistent gains in detection accuracy and robustness, especially in cases where semantic cues are misleading. Further analysis reveals that these improvements stem from a coordinated utilization of semantic and acoustic information, as opposed to simple modality aggregation.

Figures (4)

• Figure 1: Illustration of the capability gap of audio LLMs in speech deepfake detection. While audio LLMs exhibit strong semantic understanding, they struggle with reliable deepfake detection when acoustic evidence is accessed implicitly. Introducing explicit time--frequency representations reshapes acoustic evidence access, leading to more stable and reliable detection.
• Figure 2: Overview of the proposed SDD-APALLM. The framework combines raw audio and CQT spectrograms to explicitly present fine-grained acoustic evidence through time--frequency representations, facilitating speech deepfake detection within audio LLMs.
• Figure 3: Accuracy analysis across different spectrogram types and model scales. (a,b) Performance comparison between spectrogram-only and audio--spectrogram inputs for Qwen2.5-Omni-3B and Qwen2.5-Omni-7B, where $\Delta = \mathrm{ACC}(\text{Audio + Spectrogram}) - \mathrm{ACC}(\text{Spectrogram-only})$. (c,d) Accuracy difference between Qwen2.5-Omni-7B and Qwen2.5-Omni-3B under spectrogram-only and audio--spectrogram settings, computed as $\Delta = \mathrm{ACC}(7\text{B}) - \mathrm{ACC}(3\text{B})$.
• Figure 4: Attention map of SDD-APALLM during inference, illustrating token-level interactions across different input components. Tokens 1–316 correspond to the system prompt $\mathbf{T}{\mathrm{sys}}$, tokens 317–401 to the audio tokens $\mathbf{T}{\mathrm{aud}}$, tokens 402–414 to the pre-user prompt $\mathbf{T}{\mathrm{pre}}$, tokens 415–450 to the visual tokens $\mathbf{T}{\mathrm{vis}}$, and tokens beyond 450 to the post-user prompt $\mathbf{T}_{\mathrm{post}}$.