Task-Adaptive Semantic Communications with Controllable Diffusion-based Data Regeneration

TL;DR

This work tackles scalable semantic communications by enabling task-adaptive data regeneration through a diffusion-based framework. It introduces a three-phase transmitter–receiver interaction: Step 1 transmits a coarse segmentation-based semantic representation, Step 2 uses a DDPM to reconstruct and provides task-oriented feedback, and Step 3 updates the semantic embedding with attention-guided edge details for refined reconstruction. The method leverages CAM and CLIP-based attentions to identify ROIs and employs edge-map masking to create a compact, task-focused latent for refinement. Experiments on Cityscapes show superior reconstruction quality and meaningful improvements in downstream tasks (object detection and depth estimation) with favorable compression trade-offs, demonstrating the practical potential of dynamic, generative semantic transmission. The framework is compatible with existing diffusion-based semantic systems and offers a flexible path for spectrum-efficient, task-aware communications.

Abstract

Semantic communications represent a new paradigm of next-generation networking that shifts bit-wise data delivery to conveying the semantic meanings for bandwidth efficiency. To effectively accommodate various potential downstream tasks at the receiver side, one should adaptively convey the most critical semantic information. This work presents a novel task-adaptive semantic communication framework based on diffusion models that is capable of dynamically adjusting the semantic message delivery according to various downstream tasks. Specifically, we initialize the transmission of a deep-compressed general semantic representation from the transmitter to enable diffusion-based coarse data reconstruction at the receiver. The receiver identifies the task-specific demands and generates textual prompts as feedback. Integrated with the attention mechanism, the transmitter updates the semantic transmission with more details to better align with the objectives of the intended receivers. Our test results demonstrate the efficacy of the proposed method in adaptively preserving critical task-relevant information for semantic communications while preserving high compression efficiency.

Figures (5)

• Figure 1: Examples of semantic representation for autonomous driving (from left to right): (a) original image, (b) edge map, and (c) segmentation.
• Figure 2: Diagram of the proposed three-phase semantic communication: 1) Step 1: the transmitter generates a general and coarse semantic representation to provide sketch information of raw media signal; 2) Step 2: the transmitter reconstructs the original media signal via the reverse process of diffusion and also provide textual prompts as feedback; and 3) Step 3: transmitter updates the semantic representation with more task-oriented details, with which the receiver performs task-oriented reconstruction.
• Figure 3: Two types of attention schemes: 1) class activation map for classification tasks; and 2) CLIP-based attention maps for general tasks.
• Figure 4: Structure of the diffusion-based reconstruction module.
• Figure 5: Object Detection: Results of Reconstructed Images under Different Conditions: people bounded by blue boxes.