Language-oriented Semantic Communication for Image Transmission with Fine-Tuned Diffusion Model

TL;DR

A novel semantic communication framework based on a text-2-image generative model (Gen-SC) that can achieve high perceptual quality with reducing the transmitted data volume by up to 99% and is robust to wireless channel noise in terms of portrait image transmission.

Abstract

Ubiquitous image transmission in emerging applications brings huge overheads to limited wireless resources. Since that text has the characteristic of conveying a large amount of information with very little data, the transmission of the descriptive text of an image can reduce the amount of transmitted data. In this context, this paper develops a novel semantic communication framework based on a text-2-image generative model (Gen-SC). In particular, a transmitter converts the input image to textual modality data. Then the text is transmitted through a noisy channel to the receiver. The receiver then uses the received text to generate images. Additionally, to improve the robustness of text transmission over noisy channels, we designed a transformer-based text transmission codec model. Moreover, we obtained a personalized knowledge base by fine-tuning the diffusion model to meet the requirements of task-oriented transmission scenarios. Simulation results show that the proposed framework can achieve high perceptual quality with reducing the transmitted data volume by up to 99% and is robust to wireless channel noise in terms of portrait image transmission.

Figures (9)

• Figure 1: The framework of semantic communication for networks.
• Figure 2: The framework of end-to-end text transmission system.
• Figure 3: The proposed neural network structure for end-to-end text transmission system.
• Figure 4: The architecture of Stable Diffusion model.
• Figure 5: Visual results. On the left side two randomly selected samples along with text which extracted by Img2Txt model. The generated images are on the right. The SNR is 9 dB.