Unifying Speech Editing Detection and Content Localization via Prior-Enhanced Audio LLMs

TL;DR

This work tackles the challenge of detecting and localizing semantic edits in speech, which can be seamlessly integrated by end-to-end neural editing. It introduces AiEdit, a large-scale bilingual dataset generated with LLM-driven semantics and multiple neural editing methods to enable precise add/delete/modify operations. The authors then present PELM, a Prior-Enhanced Audio LLM that unifies detection and content localization as an audio question answering task, incorporating a word-level prior and a centroid-based acoustic consistency loss to mitigate forgery and semantic-priority biases. Extensive experiments on HumanEdit and AiEdit show state-of-the-art performance and robust localization, with ablations confirming the value of priors and acoustic constraints for accurate, fine-grained tampering detection. The work advances practical defenses against sophisticated speech editing by combining high-quality data, principled priors, and distribution-aware learning in large audio models.

Abstract

Speech editing achieves semantic inversion by performing fine-grained segment-level manipulation on original utterances, while preserving global perceptual naturalness. Existing detection studies mainly focus on manually edited speech with explicit splicing artifacts, and therefore struggle to cope with emerging end-to-end neural speech editing techniques that generate seamless acoustic transitions. To address this challenge, we first construct a large-scale bilingual dataset, AiEdit, which leverages large language models to drive precise semantic tampering logic and employs multiple advanced neural speech editing methods for data synthesis, thereby filling the gap of high-quality speech editing datasets. Building upon this foundation, we propose PELM (Prior-Enhanced Audio Large Language Model), the first large-model framework that unifies speech editing detection and content localization by formulating them as an audio question answering task. To mitigate the inherent forgery bias and semantic-priority bias observed in existing audio large models, PELM incorporates word-level probability priors to provide explicit acoustic cues, and further designs a centroid-aggregation-based acoustic consistency perception loss to explicitly enforce the modeling of subtle local distribution anomalies. Extensive experimental results demonstrate that PELM significantly outperforms state-of-the-art methods on both the HumanEdit and AiEdit datasets, achieving equal error rates (EER) of 0.57\% and 9.28\% (localization), respectively.

Figures (9)

• Figure 1: Spectrogram comparison across different editing samples.
• Figure 2: Self-attention heatmaps over the audio token region in the last Transformer layer.
• Figure 3: Statistical overview of the dataset composition. The bar chart (left axis) displays the total sample count for each operation type (Add, Delete, Modify) and real speech. The line plot (right axis) illustrates the total audio duration in hours for the English (blue) and Chinese (red) subsets.
• Figure 4: Distribution of Part-of-Speech (POS) tags for edited words. The pie charts illustrate the proportion of different syntactic categories targeted for editing in the Chinese (left) and English (right) subsets of our dataset.
• Figure 5: Overview of the PELM architecture, including prior-enhanced multi-modality input construction, audio LLM-based reasoning, and centroid clustering-based training objective.