Large Generative Model-assisted Talking-face Semantic Communication System

TL;DR

This study introduces a Large Generative Model-assisted Talking-face Semantic Communication (LGM-TSC) System tailored for talking-face video communication and proposes a Generative Semantic Extractor at the transmitter based on the FunASR model to convert semantically sparse talking-face videos into text with high information density.

Abstract

The rapid development of generative Artificial Intelligence (AI) continually unveils the potential of Semantic Communication (SemCom). However, current talking-face SemCom systems still encounter challenges such as low bandwidth utilization, semantic ambiguity, and diminished Quality of Experience (QoE). This study introduces a Large Generative Model-assisted Talking-face Semantic Communication (LGM-TSC) System tailored for the talking-face video communication. Firstly, we introduce a Generative Semantic Extractor (GSE) at the transmitter based on the FunASR model to convert semantically sparse talking-face videos into texts with high information density. Secondly, we establish a private Knowledge Base (KB) based on the Large Language Model (LLM) for semantic disambiguation and correction, complemented by a joint knowledge base-semantic-channel coding scheme. Finally, at the receiver, we propose a Generative Semantic Reconstructor (GSR) that utilizes BERT-VITS2 and SadTalker models to transform text back into a high-QoE talking-face video matching the user's timbre. Simulation results demonstrate the feasibility and effectiveness of the proposed LGM-TSC system.

Figures (12)

• Figure 1: The system model of the proposed SemCom system.
• Figure 2: The proposed LGM-TSC system.
• Figure 3: The architecture of the GSE.
• Figure 4: Example of disambiguation and correction by the private KB.
• Figure 5: Framework of the joint knowledge base-semantic-channel coding.