Synchronous Multi-modal Semantic Communication System with Packet-level Coding

TL;DR

The paper addresses the challenge of synchronizing multimodal semantics for video and speech over lossy networks. It proposes SyncSC, a framework that uses 3DMM-based facial semantics and speech-derived text semantics with RTP-based timing to achieve time and semantic alignment, aided by PacSC, a MAE-based packet-level forward error correction, and TextPC, a BERT-based text loss concealment. A visual-guided speech synthesis module and an image generator enable synchronized reconstruction of lip-synced speech and facial video, while loss functions and adversarial training ensure robust end-to-end performance. Experimental results on VoxCeleb and Chem demonstrate reduced transmission overhead and robust synchronous transmission under packet loss, highlighting practical benefits for talking-face transmission in video conferencing and XR scenarios.

Abstract

Although the semantic communication with joint semantic-channel coding design has shown promising performance in transmitting data of different modalities over physical layer channels, the synchronization and packet-level forward error correction of multimodal semantics have not been well studied. Due to the independent design of semantic encoders, synchronizing multimodal features in both the semantic and time domains is a challenging problem. In this paper, we take the facial video and speech transmission as an example and propose a Synchronous Multimodal Semantic Communication System (SyncSC) with Packet-Level Coding. To achieve semantic and time synchronization, 3D Morphable Mode (3DMM) coefficients and text are transmitted as semantics, and we propose a semantic codec that achieves similar quality of reconstruction and synchronization with lower bandwidth, compared to traditional methods. To protect semantic packets under the erasure channel, we propose a packet-Level Forward Error Correction (FEC) method, called PacSC, that maintains a certain visual quality performance even at high packet loss rates. Particularly, for text packets, a text packet loss concealment module, called TextPC, based on Bidirectional Encoder Representations from Transformers (BERT) is proposed, which significantly improves the performance of traditional FEC methods. The simulation results show that our proposed SyncSC reduce transmission overhead and achieve high-quality synchronous transmission of video and speech over the packet loss network.

Figures (9)

• Figure 1: The framework of synchronous semantic communication system with video-speech transmission.
• Figure 2: The architecture of video packet encoder and decoder.
• Figure 3: The architecture of text packet loss concealment.
• Figure 4: The architecture of image generator.
• Figure 5: The architecture of visual-guided speech synthesis module.