Entropy-and-Channel-Aware Adaptive-Rate Semantic Communication with MLLM-Aided Feature Compensation

TL;DR

A novel SemCom framework that incorporates an entropy-and-channel-aware adaptive rate control mechanism over MIMO Rayleigh fading channels and achieves finer-grained adaptive rate control than existing methods is proposed.

Abstract

Despite the transmission efficiency gains of semantic communication (SemCom) over traditional methods, most existing SemCom schemes still operate at a fixed transmission rate regardless of channel conditions and transmitted content, resulting in wasted resources in favorable channels and degraded performance in harsh channels. To address this issue, we propose a novel SemCom framework that incorporates an entropy-and-channel-aware adaptive rate control mechanism over MIMO Rayleigh fading channels. Specifically, we embed a joint representation of the channel state information (CSI) and the signal-to-noise ratio (SNR) into both the semantic encoder and decoder, thereby realizing channel-aware semantic coding and decoding. Moreover, the proposed method jointly exploits the CSI, the SNR, the feature maps, and their 2D entropy via two policy networks to selectively transmit only a subset of feature maps and, within each selected feature map, only a subset of symbols. Thereby, it achieves finer-grained adaptive rate control than existing methods. At the receiver, leveraging the strong visual understanding capability of multimodal large language models (MLLMs), we deploy the lightweight visual encoder (InternViT-300M) of the pre-trained InternVL3.5 model to compensate for discarded feature maps and symbols, and we fine-tune InternViT using low-rank adaptation (LoRA) for parameter-efficient training. Experimental results show that, with a carefully designed channel-aware loss function, our system automatically allocates more communication resources under poor channels to enhance task performance while reducing resource usage under favorable channels and maintaining high task performance.

Figures (13)

• Figure 1: The overall architecture of the proposed SemCom system.
• Figure 2: The network architecture of the semantic encoder.
• Figure 3: The network architecture of the channel-aware Swin Transformer block.
• Figure 4: The network architecture of the channel condition adaptive module (CCAM).
• Figure 5: The network architecture of the semantic decoder.