SoundSpring: Loss-Resilient Audio Transceiver with Dual-Functional Masked Language Modeling

TL;DR

SoundSpring addresses robust audio transmission over lossy channels while staying compatible with layered digital systems. It introduces a dual-functional architecture that uses residual vector quantization (RVQ) tokens and a masked language model (MLM) to perform both entropy modeling for compression and loss concealment for resiliency. By organizing tokens into coarse/fine layers and applying a GoS-based dependency scheme, SoundSpring achieves improved perceptual quality with reduced retransmission needs, validated on speech and music datasets under realistic packet-loss scenarios. The work demonstrates the potential of leveraging foundation-model techniques in audio communications, offering a practical pathway to scalable, real-time, resilient semantic transceivers.

Abstract

In this paper, we propose "SoundSpring", a cutting-edge error-resilient audio transceiver that marries the robustness benefits of joint source-channel coding (JSCC) while also being compatible with current digital communication systems. Unlike recent deep JSCC transceivers, which learn to directly map audio signals to analog channel-input symbols via neural networks, our SoundSpring adopts the layered architecture that delineates audio compression from digital coded transmission, but it sufficiently exploits the impressive in-context predictive capabilities of large language (foundation) models. Integrated with the casual-order mask learning strategy, our single model operates on the latent feature domain and serve dual-functionalities: as efficient audio compressors at the transmitter and as effective mechanisms for packet loss concealment at the receiver. By jointly optimizing towards both audio compression efficiency and transmission error resiliency, we show that mask-learned language models are indeed powerful contextual predictors, and our dual-functional compression and concealment framework offers fresh perspectives on the application of foundation language models in audio communication. Through extensive experimental evaluations, we establish that SoundSpring apparently outperforms contemporary audio transmission systems in terms of signal fidelity metrics and perceptual quality scores. These new findings not only advocate for the practical deployment of SoundSpring in learning-based audio communication systems but also inspire the development of future audio semantic transceivers.

Figures (13)

• Figure 1: Overview of the architecture of SoundSpring. The audio waveform is mapped to audio latent features and then multiple sequences of audio tokens are obtained by residual vector quantizer, which are emitted in order from the sequence in dark color to that in light one. A unified dual-functional MLM performs entropy modeling for efficiency at the sender and conceals the lost tokens for resiliency at the receiver. The contextual probability mass function (PMF) is leveraged for entropy coding. Given the received tokens, the MLM makes predictions utilizing the neighbouring received frames. In both functions, we mask the target tokens and replace them with a mask token. In addition, an auxiliary audio discriminator is employed in training to promote audio perceptual quality.
• Figure 2: Pipeline of MLM in SoundSpring transceiver. The RVQ audio tokens ${\boldsymbol z}$ are arranged in a two-dimensional structure, wherein the horizontal orientation denotes temporal axis, while the vertical one is the layer order starting from the bottom in dark color. After token masking, the audio tokens in one frame are projected into embeddings, which are summed and then fed into the bi-directional Transformer. The training objective of MLM is to minimize the cross entropy between the ground truth distribution of audio tokens and the predicted PMF of mask token.
• Figure 3: Two-dimensional audio token grid of $z_{t,k}$ and audio token slice grid $\mathcal{S}_{\ell,j}$. Every $T_G$ frames of tokens constitute a group of slices. The audio tokens generated by RVQ in bottom layers are assigned as coarse tokens, while the ones on the top layers are categorized into fine tokens.
• Figure 4: General dependency link of audio tokens and slice-wise dependency pattern ${\boldsymbol \Phi}$. The 2-D token slice grid is flattened into a sequence. The dependency of key slices and non-key slices are marked by orange and blue, separately.
• Figure 5: PMF of an exemplar audio token in the first fine layer, $M=1024$. Compared to a uniform prior distribution, the contextual PMF is quite unbalanced and sparse, which leads to the reduction in coding rate.