SuperCodec: A Neural Speech Codec with Selective Back-Projection Network

TL;DR

SuperCodec tackles the challenge of reconstructing high-quality speech at very low bitrates by replacing standard down- and up-sampling with Selective Down-sampling Back Projection (SDBP) and Selective Up-sampling Back Projection (SUBP) blocks, together with a selective feature fusion mechanism. The encoder comprises four SDBP modules and the decoder four SUBP modules, operating on 256-dimensional latent features at 50 Hz from 16 kHz speech, with a residual vector quantizer and a codebook of size $2^{10}$. Training uses adversarial losses with waveform and STFT discriminators, and supports bitrates of 1, 2, 3, and 6 kbps via RVQ layers, achieving state-of-the-art performance at low bitrates and outperforming Lyra V2 and Encodec in subjective and objective evaluations. Ablation studies show SUBP contributes more to performance than SDBP, and the approach offers favorable complexity and real-time factors compared to prior neural codecs. This work demonstrates a practical, high-fidelity, low-bitrate neural speech codec with potential for real-world deployment.

Abstract

Neural speech coding is a rapidly developing topic, where state-of-the-art approaches now exhibit superior compression performance than conventional methods. Despite significant progress, existing methods still have limitations in preserving and reconstructing fine details for optimal reconstruction, especially at low bitrates. In this study, we introduce SuperCodec, a neural speech codec that achieves state-of-the-art performance at low bitrates. It employs a novel back projection method with selective feature fusion for augmented representation. Specifically, we propose to use Selective Up-sampling Back Projection (SUBP) and Selective Down-sampling Back Projection (SDBP) modules to replace the standard up- and down-sampling layers at the encoder and decoder, respectively. Experimental results show that our method outperforms the existing neural speech codecs operating at various bitrates. Specifically, our proposed method can achieve higher quality reconstructed speech at 1 kbps than Lyra V2 at 3.2 kbps and Encodec at 6 kbps.

Figures (5)

• Figure 1: The architecture of SuperCodec.
• Figure 2: Network architecture of proposed Selective down and up back-projection. (a) is the selective down-sampling back-projection network(SDBP). (b) is the selective up-sampling back-projection network(SUBP). R1 and R2 are the residual blocks, respectively, consisting of the convolution layers with kernel size = 3 and dilation rates = [1, 3], and two convolution layers. S is the selective feature fusion network.
• Figure 3: Schematic for selective feature fusion block. It operates on features and performs aggregation based on self-attention. GAP is the Global Average Pooling. $\oplus$ is the element-wise summation and $\otimes$ is the element-wise product operation.
• Figure 4: MUSHRA subjective test. The indicated interval in black represents the 95% confidence interval for each score.
• Figure 5: Objective evaluation of SuperCodec at 1 kbps, 2 kbps, 3 kbps, 6 kbps. We compare our method with existing neural speech coding works using STOI, ViSQOL, and WARP-Q.