Code Drift: Towards Idempotent Neural Audio Codecs

TL;DR

The paper investigates idempotence, the stability of neural audio codecs under repeated encoding, as a practical concern for compression and iterative generative workflows. It formalizes idempotence as $f(x) \approx f(f(x))$ for a codec $f$ and measures stability across audio quality metrics $PESQ$ and $SI-SDR$, along with token-space metrics like token match rate and codebook entropy, over up to 25 encodings. To improve idempotence, it introduces three regularizers—$L_{enc}$, $L_{proj}$, and $L_{code}$—and finds that fine-tuning with $L_{code}$ yields the strongest gains while preserving perceptual quality. Downstream, it shows that models trained on idempotence-enhanced codecs perform comparably to those trained on the base codec in a conditional generative task, suggesting practical benefits for compression and iterative audio generation without sacrificing performance.

Abstract

Neural codecs have demonstrated strong performance in high-fidelity compression of audio signals at low bitrates. The token-based representations produced by these codecs have proven particularly useful for generative modeling. While much research has focused on improvements in compression ratio and perceptual transparency, recent works have largely overlooked another desirable codec property -- idempotence, the stability of compressed outputs under multiple rounds of encoding. We find that state-of-the-art neural codecs exhibit varied degrees of idempotence, with some degrading audio outputs significantly after as few as three encodings. We investigate possible causes of low idempotence and devise a method for improving idempotence through fine-tuning a codec model. We then examine the effect of idempotence on a simple conditional generative modeling task, and find that increased idempotence can be achieved without negatively impacting downstream modeling performance -- potentially extending the usefulness of neural codecs for practical file compression and iterative generative modeling workflows.

Figures (3)

• Figure 1: Effect of idempotent training: audio quality (in PESQ and SI-SDR), average codebook use, and match rate between tokens encoded at the $n^{\mathrm{th}}$ and $(n + 1)^{\mathrm{th}}$ iterations as a function of encoding iterations for our DAC reproduction and fine-tuned variants with idempotence losses described in Section \ref{['sec:encouraging']}. Fine-tuning with $\mathcal{L}_{code}$ results in the highest overall idempotence through 25 encoding iterations, while preserving codebook use.
• Figure 2: Audio and code stability over encodings: audio quality (in PESQ and SI-SDR) of selected neural codecs by encoding iteration, average codebook use by encoding iteration, and match rate between tokens encoded at the $n^{\mathrm{th}}$ and $(n + 1)^{\mathrm{th}}$ iterations.
• Figure 3: Phase sensitivity correlates with idempotence: For each selected codec, we compute the token match rate between encodings of audio inputs under time shifts between $\pm 2$ms. We then compute the correlation between average observed match rate and PESQ score after 25 encoding iterations. In general, codecs with higher phase sensitivity (as indicated by lower match rates) exhibit higher idempotence (as indicated by higher PESQ scores).