Mentor-Telemachus Bond: Transferring Knowledge in Semantic Communication via Contrastive Learning

TL;DR

This work addresses the scalability and generalization gaps in semantic communication by decoupling parametric knowledge ($S_e$, $L_e$) from non-parametric knowledge stored in shared/private bases ($SKB$, $PKB$) and transferring it via contrastive learning. It introduces CRLSC, a two-stage framework where a server-side pre-trained model (e.g., CLIP) constructs a shared knowledge base encoded as vectors with Product Quantization, guiding local encoders to learn representations that populate private knowledge bases for task-specific decoding. Stage 2 fine-tunes decoders (e.g., VQ-VAE) on downstream tasks, resulting in improved generalization and robustness across datasets while reducing local computation and data annotation needs. The proposed approach enables scalable semantic communication across large, heterogeneous networks by facilitating knowledge transfer and adaptable decoding, with practical impact on real-world AI-driven communication systems.

Abstract

Encoder, decoder and knowledge base are three major components for semantic communication. Recent advances have achieved significant progress in the encoder-decoder design. However, there remains a considerable gap in the construction and utilization of knowledge base, which plays important roles in establishing consensus among communication participants through knowledge transferring and sharing. Current knowledge base designs typically involve complex structures, which lead to significant computational overheads and heavy reliance on manually annotated datasets, making it difficult to adapt to existing encoder-decoder models. Hence, without knowledge transferring and sharing within the network results in poor generalization of encoder-decoder. This necessitates model training for specific tasks and datasets, significantly limiting the scalability of semantic communication systems to larger networks. To address these challenges, we propose an innovative Contrastive Representations Learning based Semantic Communication Framework (CRLSC). In CRLSC, the server-side pre-trained large model utilizes large-scale public datasets to construct shared knowledge base. Local-side encoders in terminal devices conduct training guided by shared knowledge base. These trained encoders can then build private knowledge bases from private datasets and fine-tune decoders for specific tasks. This simple and effective approach can facilitate the knowledge transferring across large-scale heterogeneous networks.

Figures (6)

• Figure 1: The comparison between conventional communication and semantic communication. The upper branch represents conventional communication, while the lower branch is semantic communication filtering out redundant information from the raw data.
• Figure 2: Contrastive Representations Learning based Semantic Communication Framework
• Figure 3: Main Learning Process
• Figure 4: Data Reconstruction on Animal Dataset. Odd rows represent reconstructed images, while even rows represent original images.
• Figure 5: Data Reconstruction on Cleba Dataset. Odd rows represent reconstructed images, while even rows represent original images.