Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large AI Model-Based Semantic Communications

Feibo Jiang, Yubo Peng, Li Dong, Kezhi Wang, Kun Yang, Cunhua Pan, Xiaohu You

TL;DR

This work addresses semantic communication (SC) for image data by introducing a large AI model–based knowledge base (LAM-KB) framework (LAM-SC). It leverages a Segment Anything Model (SAM)–based KB (SKB) to segment images into semantic objects, an attention-based semantic integration (ASI) module to weight and fuse salient segments, and an adaptive semantic compression (ASC) mechanism to mask redundant semantic features before channel encoding. Training combines human-guided ASI supervision, crossover training of encoders/decoders, and joint ASC optimization, validated on VOC2012 and COCO2017, showing lower system loss and higher SSIM with substantially reduced bit usage (LAM-SC semantic features ~8,960 bits vs 21,632 bits for traditional SC and 49,152 bits for raw images, ~55% data relative to SC). The results demonstrate the feasibility of LAM-based KBs to enhance image-based SC and point to open issues like latency, energy efficiency, explainability, and privacy for future SC paradigms.

Abstract

Semantic communication (SC) is an emerging intelligent paradigm, offering solutions for various future applications like metaverse, mixed reality, and the Internet of Everything. However, in current SC systems, the construction of the knowledge base (KB) faces several issues, including limited knowledge representation, frequent knowledge updates, and insecure knowledge sharing. Fortunately, the development of the large AI model (LAM) provides new solutions to overcome the above issues. Here, we propose a LAM-based SC framework (LAM-SC) specifically designed for image data, where we first apply the segment anything model (SAM)-based KB (SKB) that can split the original image into different semantic segments by universal semantic knowledge. Then, we present an attention-based semantic integration (ASI) to weigh the semantic segments generated by SKB without human participation and integrate them as the semantic aware image. Additionally, we propose an adaptive semantic compression (ASC) encoding to remove redundant information in semantic features, thereby reducing communication overhead. Finally, through simulations, we demonstrate the effectiveness of the LAM-SC framework and the possibility of applying the LAM-based KB in future SC paradigms.

Large AI Model-Based Semantic Communications

TL;DR

This work addresses semantic communication (SC) for image data by introducing a large AI model–based knowledge base (LAM-KB) framework (LAM-SC). It leverages a Segment Anything Model (SAM)–based KB (SKB) to segment images into semantic objects, an attention-based semantic integration (ASI) module to weight and fuse salient segments, and an adaptive semantic compression (ASC) mechanism to mask redundant semantic features before channel encoding. Training combines human-guided ASI supervision, crossover training of encoders/decoders, and joint ASC optimization, validated on VOC2012 and COCO2017, showing lower system loss and higher SSIM with substantially reduced bit usage (LAM-SC semantic features ~8,960 bits vs 21,632 bits for traditional SC and 49,152 bits for raw images, ~55% data relative to SC). The results demonstrate the feasibility of LAM-based KBs to enhance image-based SC and point to open issues like latency, energy efficiency, explainability, and privacy for future SC paradigms.

Abstract

Semantic communication (SC) is an emerging intelligent paradigm, offering solutions for various future applications like metaverse, mixed reality, and the Internet of Everything. However, in current SC systems, the construction of the knowledge base (KB) faces several issues, including limited knowledge representation, frequent knowledge updates, and insecure knowledge sharing. Fortunately, the development of the large AI model (LAM) provides new solutions to overcome the above issues. Here, we propose a LAM-based SC framework (LAM-SC) specifically designed for image data, where we first apply the segment anything model (SAM)-based KB (SKB) that can split the original image into different semantic segments by universal semantic knowledge. Then, we present an attention-based semantic integration (ASI) to weigh the semantic segments generated by SKB without human participation and integrate them as the semantic aware image. Additionally, we propose an adaptive semantic compression (ASC) encoding to remove redundant information in semantic features, thereby reducing communication overhead. Finally, through simulations, we demonstrate the effectiveness of the LAM-SC framework and the possibility of applying the LAM-based KB in future SC paradigms.
Paper Structure (32 sections, 6 figures)

This paper contains 32 sections, 6 figures.

Table of Contents

  1. Introduction
  2. Composition of a Universal KB in SC Systems
  3. Issues about Current KB Schemes in SC Systems
  4. Our Contributions
  5. LAMs-Based KBs in SC systems
  6. Advantages of LAMs-Based KBs
  7. Design Suggestions of LAMs in SC Systems
  8. GPT-Based KBs
  9. SAM-Based KBs
  10. WavLM-Based KBs
  11. Architecture of LAM-SC Framework
  12. Introduction to LAM-SC Framework
  13. KB Construction and Semantic Segmentation
  14. Attention-Based Semantic Integration
  15. Semantic Adaptive Encoding and Channel Encoding
  16. ...and 17 more sections

Figures (6)

  • Figure 1: Implementation of LAMs-based KBs in different SC models.
  • Figure 2: The illustration of the proposed LAM-SC framework.
  • Figure 3: The training process of the proposed LAM-SC framework.
  • Figure 4: The system loss results of different schemes.
  • Figure 5: The SSIM results of different schemes on VOC2012 dataset.
  • ...and 1 more figures