Scene Understanding Enabled Semantic Communication with Open Channel Coding

TL;DR

This work tackles semantic communication for 6G by integrating scene-graph–based structured semantics with open-channel coding and LLM-assisted decoding. The proposed OpenSC framework enables adaptive, knowledge-agnostic transmission and improves VQA performance under varying channel conditions while reducing transmission redundancy. Key contributions include selective scene-graph encoding, dynamic open-channel coding, LLM-enabled semantic enhancements, JSON-formatted outputs, and a vector-database–assisted prompt strategy, all demonstrated to boost semantic understanding and efficiency. The results indicate meaningful gains in robustness and bandwidth efficiency, underscoring OpenSC's potential for scalable multimodal semantic communication in real-world 6G networks.

Abstract

As communication systems transition from symbol transmission to conveying meaningful information, sixth-generation (6G) networks emphasize semantic communication. This approach prioritizes high-level semantic information, improving robustness and reducing redundancy across modalities like text, speech, and images. However, traditional semantic communication faces limitations, including static coding strategies, poor generalization, and reliance on task-specific knowledge bases that hinder adaptability. To overcome these challenges, we propose a novel system combining scene understanding, Large Language Models (LLMs), and open channel coding, named \textbf{OpenSC}. Traditional systems rely on fixed domain-specific knowledge bases, limiting their ability to generalize. Our open channel coding approach leverages shared, publicly available knowledge, enabling flexible, adaptive encoding. This dynamic system reduces reliance on static task-specific data, enhancing adaptability across diverse tasks and environments. Additionally, we use scene graphs for structured semantic encoding, capturing object relationships and context to improve tasks like Visual Question Answering (VQA). Our approach selectively encodes key semantic elements, minimizing redundancy and improving transmission efficiency. Experimental results show significant improvements in both semantic understanding and efficiency, advancing the potential of adaptive, generalizable semantic communication in 6G networks.

Figures (9)

• Figure 1: The Comparison of a semantic communication framework with visual scene understanding, incorporating semantic coding and knowledge-agnostic channel coding, with traditional semantic communication approaches yang2022semantic.
• Figure 2: The overall framework of scene understanding-enabled semantic communication with open-channel coding, consisting of three modules: semantic encoding, open-channel coding (encoding and decoding), and semantic decoding. The section marked as "Flame" represents the core contribution of this paper, where we propose a structured semantic encoding method to address challenges in visual information, scene understanding, and interpretability. By leveraging LLMs, we address the problem of knowledge-agnostic open-channel coding.
• Figure 3: Recall performance evaluation under different SNR levels in an AWGN channel using a 16QAM modulation scheme. The MJCMSC system represents our constructed JCMSC multimodal system, where 5-bit + RS Huffman + RS JPEG + LDPC all use 16QAM. The performance is evaluated for four types of questions: Category, Quantity, Location, and Relationship.
• Figure 4: F1-score evaluation for four types of questions (Category, Quantity, Location, Relationship) under different SNR levels in an AWGN channel using a 16QAM modulation scheme.
• Figure 5: Recall performance evaluation for four types of questions (Category, Quantity, Location, Relationship) under different SNR levels in an AWGN channel, using BPSK, 4QAM, and 16QAM modulation schemes.