The Equalizer: Introducing Shape-Gain Decomposition in Neural Audio Codecs

TL;DR

The Equalizer addresses the sensitivity of neural audio codecs to global input level by enforcing a frame-wise shape–gain decomposition prior to encoding and a corresponding deequalization after decoding. By normalizing each short-term frame to a fixed energy, the NAC processes only the normalized shape, while a separate mu-law scalar quantizer handles the frame gain, reducing codebook redundancy and enabling more efficient rate-distortion performance. Experimental results on 16 kHz speech (LibriSpeech data) show that the Equalizer achieves gain-invariant reconstruction with substantially better PESQ, STOI, and SI-SDR across bitrates, often matching or exceeding the baseline at higher bitrates while using far smaller codebooks, and even achieving large bitrate reductions with modest quality loss. The approach is modular and can be applied to any NAC, offering improved robustness to input variability, better interpretability, and potential complexity reductions in practical deployments.

Abstract

Neural audio codecs (NACs) typically encode the short-term energy (gain) and normalized structure (shape) of speech/audio signals jointly within the same latent space. As a result, they are poorly robust to a global variation of the input signal level in the sense that such variation has strong influence on the embedding vectors at the output of the encoder and their quantization. This methodology is inherently inefficient, leading to codebook redundancy and suboptimal bitrate-distortion performance. To address these limitations, we propose to introduce shape-gain decomposition, widely used in classical speech/audio coding, into the NAC framework. The principle of the proposed Equalizer methodology is to decompose the input signal -- before the NAC encoder -- into gain and normalized shape vector on a short-term basis. The shape vector is processed by the NAC, while the gain is quantized with scalar quantization and transmitted separately. The output (decoded) signal is reconstructed from the normalized output of the NAC and the quantized gain. Our experiments conducted on speech signals show that this general methodology, easily applicable to any NAC, enables a substantial gain in bitrate-distortion performance, as well as a massive reduction in complexity.

Figures (5)

• Figure 1: Effect of global input gain variation on BigCodec xin2024bigcodec, SpeechTokenizer zhang2023speechtokenizer and DAC kumar2023high latent representations. Top: Average embedding norm (normalized to 0dB). Middle: Average cosine similarity with the 0dB reference embedding. Bottom: Discrete code stability after quantization of the embedding vectors. The vertical dashed line marks 0dB, the reference gain.
• Figure 2: Architecture of the proposed Equalizer neural audio coding framework based on shape-gain decomposition. Block 1: The input signal $\mathbf{s}$ is decomposed into a temporal gain envelope and an equalized waveform $\bar{\mathbf{s}}$ using short-term analysis, normalization, and OLA synthesis. Block 2: The resulting successive shape vectors are processed by a NAC applying vector quantization on the corresponding embedding vectors. Block 3: In parallel, the gain is quantized with scalar quantization (typically $\mu$-law). Block 4: The decoded equalized output waveform and the quantized gain are used to generate the final output waveform using again short-term analysis-synthesis with OLA.
• Figure 3: Performance of the proposed Equalizer method and of the baselines, as a function of input gain $\alpha$, for 4 different values of the codebook size $C$. Each score is averaged over $\approx 5.4$ h of test signals from the Librispeech-test dataset. The proposed Equalizer method utilizing the shape-gain decomposition (solid lines) exhibits performance invariance, whereas the ablated baseline (dashed lines) degrades significantly as the gain deviates from 0 dB. Moreover, the proposed method consistently outperforms the ablated baseline even at $0$ dB.
• Figure 4: Performance of the proposed Equalizer method and of the baselines, as a function of the (total) bitrate. The scores are averaged over the entire test dataset and over a gain range of $\pm$12 dB. The Equalizer (solid blue lines) consistently outperforms the ablated baseline (dashed orange lines) and SpeechTokenizer (purple triangle).
• Figure 5: Impact of RVQ depth on coding quality. The scores are averaged over the entire test dataset and over a gain range of $\pm$12 dB.