Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Localizing Speech Deepfakes Beyond Transitions via Segment-Aware Learning

Yuchen Mao, Wen Huang, Yanmin Qian

TL;DR

This work tackles partial speech deepfake localization by shifting focus from boundary artifacts to the intrinsic structure of entire segments. It introduces Segment Positional Labeling (SPL) as a fine-grained, multi-label supervision and Cross-Segment Mixing (CSM) as a data augmentation strategy, both integrated into a Segment-Aware Learning (SAL) framework. Across PS, HAD, and LPS datasets, SAL demonstrates strong in-domain and cross-domain performance, improves localization in non-boundary regions, and reduces shortcut learning, as illustrated by Grad-CAM analyses. The approach offers robust, generalizable tooling for precise deepfake localization with practical implications for anti-spoofing systems.

Abstract

Localizing partial deepfake audio, where only segments of speech are manipulated, remains challenging due to the subtle and scattered nature of these modifications. Existing approaches typically rely on frame-level predictions to identify spoofed segments, and some recent methods improve performance by concentrating on the transitions between real and fake audio. However, we observe that these models tend to over-rely on boundary artifacts while neglecting the manipulated content that follows. We argue that effective localization requires understanding the entire segments beyond just detecting transitions. Thus, we propose Segment-Aware Learning (SAL), a framework that encourages models to focus on the internal structure of segments. SAL introduces two core techniques: Segment Positional Labeling, which provides fine-grained frame supervision based on relative position within a segment; and Cross-Segment Mixing, a data augmentation method that generates diverse segment patterns. Experiments across multiple deepfake localization datasets show that SAL consistently achieves strong performance in both in-domain and out-of-domain settings, with notable gains in non-boundary regions and reduced reliance on transition artifacts. The code is available at https://github.com/SentryMao/SAL.

Localizing Speech Deepfakes Beyond Transitions via Segment-Aware Learning

TL;DR

This work tackles partial speech deepfake localization by shifting focus from boundary artifacts to the intrinsic structure of entire segments. It introduces Segment Positional Labeling (SPL) as a fine-grained, multi-label supervision and Cross-Segment Mixing (CSM) as a data augmentation strategy, both integrated into a Segment-Aware Learning (SAL) framework. Across PS, HAD, and LPS datasets, SAL demonstrates strong in-domain and cross-domain performance, improves localization in non-boundary regions, and reduces shortcut learning, as illustrated by Grad-CAM analyses. The approach offers robust, generalizable tooling for precise deepfake localization with practical implications for anti-spoofing systems.

Abstract

Localizing partial deepfake audio, where only segments of speech are manipulated, remains challenging due to the subtle and scattered nature of these modifications. Existing approaches typically rely on frame-level predictions to identify spoofed segments, and some recent methods improve performance by concentrating on the transitions between real and fake audio. However, we observe that these models tend to over-rely on boundary artifacts while neglecting the manipulated content that follows. We argue that effective localization requires understanding the entire segments beyond just detecting transitions. Thus, we propose Segment-Aware Learning (SAL), a framework that encourages models to focus on the internal structure of segments. SAL introduces two core techniques: Segment Positional Labeling, which provides fine-grained frame supervision based on relative position within a segment; and Cross-Segment Mixing, a data augmentation method that generates diverse segment patterns. Experiments across multiple deepfake localization datasets show that SAL consistently achieves strong performance in both in-domain and out-of-domain settings, with notable gains in non-boundary regions and reduced reliance on transition artifacts. The code is available at https://github.com/SentryMao/SAL.
Paper Structure (10 sections, 3 equations, 3 figures, 4 tables)

This paper contains 10 sections, 3 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Segment-Aware Learning
  3. Segment Positional Labeling
  4. Cross-Segment Mixing
  5. Experiments and Analysis
  6. Experimental Settings
  7. Performance Comparison
  8. Ablation Study
  9. Visualization and Analysis
  10. Conclusion

Figures (3)

  • Figure 1: Prediction scores and Grad-CAM visualizations (deeper color indicates higher values) for CON_E_0000096.wav in PartialSpoof evaluation set, comparing the baseline frame-level detection system (top) and our segment-aware learning system (bottom).
  • Figure 2: Overview of the proposed Segment-Aware Learning (SAL) framework. Arrows in different colors indicate the distinction between the baseline Frame-Level Detection (FLD) pipeline and the proposed SAL strategy.
  • Figure 3: Visualization of model performance on different positions. (a) shows that middle-segments are the most common case. (b) demonstrates that our SAL model significantly reduces errors on this specific category compared to the FLD model.