Do Neural Codecs Generalize? A Controlled Study Across Unseen Languages and Non-Speech Tasks

TL;DR

This work systematically examines the generalization capabilities of neural audio codecs (NACs) along three axes: unseen languages during pre-training, transfer to non-speech domains from speech-only pre-training, and the effect of including non-speech data in pre-training. The authors train NACs from scratch under strictly controlled data coverages using a GAN-based framework with an encoder $\\mathcal{E}$, a quantizer $\\mathcal{Q}$, a codebook $\\mathcal{B}$, and a decoder $\\mathcal{D}$, generating discrete codes $\\mathbf{C}=\\mathcal{Q}(\\mathcal{E}(Y),\\mathcal{B})$ and reconstructed signals $\\hat{Y}=\\mathcal{D}(\\mathbf{C},\\mathcal{B})$. They evaluate on 11 metrics across signal reconstruction (speech and non-speech) and a downstream TTS task, using three pre-training data coverages: English, Multilingual, and Multilingual+Audio. The results show that NACs can generalize to languages not seen during pre-training; speech-only pre-training underperforms on non-speech tasks; and including non-speech data improves non-speech performance while preserving speech performance, suggesting joint pre-training as a robust strategy for broad-domain NAC applications. These findings inform data curation and pre-training design for NACs and motivate extending evaluations to additional architectures and downstream tasks beyond TTS.

Abstract

This paper investigates three crucial yet underexplored aspects of the generalization capabilities of neural audio codecs (NACs): (i) whether NACs can generalize to unseen languages during pre-training, (ii) whether speech-only pre-trained NACs can effectively generalize to non-speech applications such as environmental sounds, music, and animal vocalizations, and (iii) whether incorporating non-speech data during pre-training can improve performance on both speech and non-speech tasks. Existing studies typically rely on off-the-shelf NACs for comparison, which limits insight due to variations in implementation. In this work, we train NACs from scratch using strictly controlled configurations and carefully curated pre-training data to enable fair comparisons. We conduct a comprehensive evaluation of NAC performance on both signal reconstruction quality and downstream applications using 11 metrics. Our results show that NACs can generalize to unseen languages during pre-training, speech-only pre-trained NACs exhibit degraded performance on non-speech tasks, and incorporating non-speech data during pre-training improves performance on non-speech tasks while maintaining comparable performance on speech tasks.

Figures (1)

• Figure 1: Overview of our three key research questions.