Multi-Task Semantic Communication With Graph Attention-Based Feature Correlation Extraction

TL;DR

This work tackles the challenge of bandwidth-constrained, multi-task semantic communication by introducing a Graph Attention Inter-block (GAI) module that enriches encoder features. The GAI constructs a graph from intermediate encoder block outputs, applies a Graph Attention Layer to refine node representations, and uses a Relation Mapping Layer to generate task-specific weights for each block, enabling tailored feature transmission $\boldsymbol{z}_t$ for every task. Empirical results across CityScapes, NYU v2, TaskonomyTiny, Oxford-IIIT Pet, and MVSA show systematic gains over strong baselines, including up to $11.4\%$ improvement under extreme bandwidth constraints ($R=\frac{1}{12}$) and robustness across a broad range of SNRs. The approach highlights the importance of modeling inter-feature correlations for efficient multi-task transmission and offers a scalable path toward practical, task-aware semantic communication in bandwidth-limited environments.

Abstract

Multi-task semantic communication can serve multiple learning tasks using a shared encoder model. Existing models have overlooked the intricate relationships between features extracted during an encoding process of tasks. This paper presents a new graph attention inter-block (GAI) module to the encoder/transmitter of a multi-task semantic communication system, which enriches the features for multiple tasks by embedding the intermediate outputs of encoding in the features, compared to the existing techniques. The key idea is that we interpret the outputs of the intermediate feature extraction blocks of the encoder as the nodes of a graph to capture the correlations of the intermediate features. Another important aspect is that we refine the node representation using a graph attention mechanism to extract the correlations and a multi-layer perceptron network to associate the node representations with different tasks. Consequently, the intermediate features are weighted and embedded into the features transmitted for executing multiple tasks at the receiver. Experiments demonstrate that the proposed model surpasses the most competitive and publicly available models by 11.4% on the CityScapes 2Task dataset and outperforms the established state-of-the-art by 3.97% on the NYU V2 3Task dataset, respectively, when the bandwidth ratio of the communication channel (i.e., compression level for transmission over the channel) is as constrained as 1 12 .

Figures (11)

• Figure 1: The proposed multi-task semantic communication system model, where the encoder extracts features, which are transmitted through the channel to the decoders for multiple simultaneous tasks.
• Figure 2: The proposed GAI module, which involves generating encoded data $\mathbf{z}_t$ for task $t$ within the GAI module. This process begins with inputs from the output of each basic encoder block. The procedure comprises (i) a Feature Transformation Layer, (ii) a Graph Attention Layer, and (iii) a Relation Mapping Layer. The Feature Transformation Layer unifies each block's feature size, and the Graph Attention Layer updates the node representation to interact with the other nodes. The Relation Mapping Layer generates task-node weights specific to each task, denoted as $\mathbf{e}_{i,t}$. $\mathbf{e}_{i,t}$ is merged with $\mathbf{K}_i$ to obtain the task-specific transmission representation $\mathbf{z}_t$ for each respective task.
• Figure 3: Graph Attention Layer: This diagram illustrates the iterative process of updating representation for the $i$-th node, specifically focusing on the $m$-th update iteration. Each node undergoes a total of $M$ iterations for information updating.
• Figure 4: Improvement of GAI over Ada under different SNR conditions in the CityScapes 2Task and NYU v2 3Task transmission.
• Figure 5: Improvement of GAI over Ada under different SNR conditions in the CityScapes 2Task and NYU v2 3Task transmission over fading channels.