Weakly-supervised Audio Temporal Forgery Localization via Progressive Audio-language Co-learning Network

TL;DR

This work addresses weakly supervised audio temporal forgery localization (wATFL) by introducing LOCO, a progressive audio-language co-learning network. LOCO combines an Audio-Language Co-learning (A2LC) module with a Progressive Refinement Strategy (PRS) to learn forgery-aware features using only utterance-level labels, avoiding expensive frame-level annotations. The A2LC module uses a Temporal Forgery Attention (TFA) and a Prompt-Enhanced Forgery Feature (PFF) adapter to align temporal and global semantics, while PRS uses semantic contrastive learning and pseudo labels to iteratively refine frame-level localization. Extensive experiments on HAD, LAV-DF, and AV-Deepfake-1M demonstrate state-of-the-art performance in both frame-level detection and temporal localization under weak supervision, highlighting LOCO’s practical potential for defending against partial audio forgeries.

Abstract

Audio temporal forgery localization (ATFL) aims to find the precise forgery regions of the partial spoof audio that is purposefully modified. Existing ATFL methods rely on training efficient networks using fine-grained annotations, which are obtained costly and challenging in real-world scenarios. To meet this challenge, in this paper, we propose a progressive audio-language co-learning network (LOCO) that adopts co-learning and self-supervision manners to prompt localization performance under weak supervision scenarios. Specifically, an audio-language co-learning module is first designed to capture forgery consensus features by aligning semantics from temporal and global perspectives. In this module, forgery-aware prompts are constructed by using utterance-level annotations together with learnable prompts, which can incorporate semantic priors into temporal content features dynamically. In addition, a forgery localization module is applied to produce forgery proposals based on fused forgery-class activation sequences. Finally, a progressive refinement strategy is introduced to generate pseudo frame-level labels and leverage supervised semantic contrastive learning to amplify the semantic distinction between real and fake content, thereby continuously optimizing forgery-aware features. Extensive experiments show that the proposed LOCO achieves SOTA performance on three public benchmarks.

Figures (5)

• Figure 1: (a) The pipeline of partial forgery audio manipulation, which poses an urgent need to study advanced detection and localization methods. (b) Differences in training between existing countermeasures and the proposed LOCO, highlighting the challenges of weakly-supervised learning from utterance-level labels.
• Figure 2: The network structure of the proposed LOCO. It incorporates an A2LC module to capture forgery consensus features by aligning semantics from temporal and global perspectives, followed by a PRS to amplify the semantic distinction between real and fake content.
• Figure 3: Qualitative prediction results based on LOCO's prediction scores where red region indicates forgery segments.
• Figure 4: Qualitative localization comparison with ground-truth annotations and comparison methods in terms of predicted proposals.
• Figure 5: t-SNE visualization of latent features with different loss functions. Green and red dots denote real and forgery frames.