Not All Deepfakes Are Created Equal: Triaging Audio Forgeries for Robust Deepfake Singer Identification

TL;DR

Not All Deepfakes Are Created Equal: Triaging Audio Forgeries for Robust Deepfake Singer Identification addresses the problem of identifying a singer in both authentic and synthetic singing voices. It introduces a two-stage pipeline where a discriminator $\mathcal{D}$ (Light CNN) filters out low-quality forgeries and a singer-identity model $\mathcal{S}$ (ecapa-tdnn) trained on authentic data performs identity matching via cosine-distance embeddings. The approach demonstrates performance gains over baselines across multiple datasets (ctrsvdd, wildsvdd) and reveals that high-quality deepfakes pose the greatest challenge, with the discriminator helping to ensure $\mathcal{S}$ operates on recognizable vocal likenesses. These findings offer a scalable defense for artist likeness rights by combining deepfake screening with authentic-data-trained singer identification.

Abstract

The proliferation of highly realistic singing voice deepfakes presents a significant challenge to protecting artist likeness and content authenticity. Automatic singer identification in vocal deepfakes is a promising avenue for artists and rights holders to defend against unauthorized use of their voice, but remains an open research problem. Based on the premise that the most harmful deepfakes are those of the highest quality, we introduce a two-stage pipeline to identify a singer's vocal likeness. It first employs a discriminator model to filter out low-quality forgeries that fail to accurately reproduce vocal likeness. A subsequent model, trained exclusively on authentic recordings, identifies the singer in the remaining high-quality deepfakes and authentic audio. Experiments show that this system consistently outperforms existing baselines on both authentic and synthetic content.

Figures (2)

• Figure 1: Proposed two-stage pipeline for singer identification. Stage 1 ($\mathcal{D}$) filters low-quality deepfakes. Tracks classified as authentic proceed to Stage 2, where singer identity is determined by nearest neighbor search using cosine distance of extracted embeddings ($e$).
• Figure 2: Confusion matrices of the considered singing voice discriminator $\mathcal{D}$ across the four different datasets. False Positive Rate (FPR) (i.e., authentic tracks misclassified as deepfakes) is exceptionally low across all datasets (see top right corners of the confusion matrices). This means that when $\mathcal{D}$ identifies a track as authentic, it is highly reliable.