SONICS: Synthetic Or Not -- Identifying Counterfeit Songs

TL;DR

This work tackles the rising challenge of end-to-end AI-generated songs by introducing the SONICS dataset, a large-scale collection of real and synthetic songs with lyrics and long durations. It demonstrates that modeling long-range temporal dependencies is crucial for authentic fake-song detection and presents SpecTTTra, a spectro-temporal transformer that tokenizes spectrograms into separate temporal and spectral streams to achieve efficiency without sacrificing accuracy. Across AI benchmarks and efficiency tests, SpecTTTra variants outperform traditional CNN and ViT baselines, particularly on long audio, and a human benchmark confirms AI methods' superior detection capability. The study lays the groundwork for robust, scalable end-to-end fake-song detection and provides rich resources for future research and benchmarking in music forensics.

Abstract

The recent surge in AI-generated songs presents exciting possibilities and challenges. These innovations necessitate the ability to distinguish between human-composed and synthetic songs to safeguard artistic integrity and protect human musical artistry. Existing research and datasets in fake song detection only focus on singing voice deepfake detection (SVDD), where the vocals are AI-generated but the instrumental music is sourced from real songs. However, these approaches are inadequate for detecting contemporary end-to-end artificial songs where all components (vocals, music, lyrics, and style) could be AI-generated. Additionally, existing datasets lack music-lyrics diversity, long-duration songs, and open-access fake songs. To address these gaps, we introduce SONICS, a novel dataset for end-to-end Synthetic Song Detection (SSD), comprising over 97k songs (4,751 hours) with over 49k synthetic songs from popular platforms like Suno and Udio. Furthermore, we highlight the importance of modeling long-range temporal dependencies in songs for effective authenticity detection, an aspect entirely overlooked in existing methods. To utilize long-range patterns, we introduce SpecTTTra, a novel architecture that significantly improves time and memory efficiency over conventional CNN and Transformer-based models. For long songs, our top-performing variant outperforms ViT by 8% in F1 score, is 38% faster, and uses 26% less memory, while also surpassing ConvNeXt with a 1% F1 score gain, 20% speed boost, and 67% memory reduction.

Figures (11)

• Figure 1: End-to-end pipeline of Full Fake song generation process. Here a random combination of topic, genre, and mood is utilized to create fake lyrics and fake song styles by prompt engineering LLM GPT-4o. The generated lyrics and styles are utilized to create synthetic songs through audio-generative models from Suno and Udio.
• Figure 2: End-to-end pipeline of Mostly Fake song generation process. Firstly, lyrics features are extracted using an LLM from real song lyrics, which in turn is utilized to create fake lyrics through a second call to the LLM GPT-4o. Secondly, style information is extracted from real songs using multimodal Generative model Gemini 1.5 Flash. Finally, the generated lyrics and styles are utilized to create synthetic songs through audio-generative models from Suno and Udio.
• Figure 3: Proposed Spectro-Temporal Tokens Transformer (SpecTTTra) model. First, the input mel-spectrogram undergoes separate temporal and spectral slicing to generate corresponding clips, which are tokenized into temporal and spectral tokens using separate tokenizers. Next, separate positional embeddings are added to these tokens, which are then passed through a Transformer encoder. The resulting globally contextualized features are then average pooled and finally passed to the classifier.
• Figure 4: Comparison of the number of tokens generated by the ViT model and the three SpecTTTra variants ($\alpha$, $\beta$, and $\gamma$) as a function of the number of time frames.
• Figure 5: Long Context Correlation. The red highlighted regions in the spectrogram and raw audio indicate repetition of the same verse ("Near, far, wherever you are, I believe that the heart does go on"), rhythms, and music. Such consistency is a characteristic of real songs and can be challenging for synthetic generation methods to replicate.