Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TACO: Rethinking Semantic Communications with Task Adaptation and Context Embedding

Achintha Wijesinghe, Weiwei Wang, Suchinthaka Wanninayaka, Songyang Zhang, Zhi Ding

TL;DR

This work tackles the challenge of semantic communications under evolving receiver goals by separating context information from task-specific content. It introduces TACO, a framework that embeds both context and GO content in a shared VQ-VAE latent space, uses GradCAM to guide ROI-based information selection, and leverages a Local Semantic Feedback loop for adaptive rate control and efficient latent fusion. The approach enables rapid adaptation to new downstream tasks with minimal additional bandwidth, improving reconstruction quality, task performance, and latency across image reconstruction, classification, and object detection benchmarks, while outperforming diffusion-based and conventional baselines. The findings suggest significant practical impact for bandwidth-efficient SemCom in dynamic multi-task settings and point to future work on channel noise and context-content compression techniques.

Abstract

Recent advancements in generative artificial intelligence have introduced groundbreaking approaches to innovating next-generation semantic communication, which prioritizes conveying the meaning of a message rather than merely transmitting raw data. A fundamental challenge in semantic communication lies in accurately identifying and extracting the most critical semantic information while adapting to downstream tasks without degrading performance, particularly when the objective at the receiver may evolve over time. To enable flexible adaptation to multiple tasks at the receiver, this work introduces a novel semantic communication framework, which is capable of jointly capturing task-specific information to enhance downstream task performance and contextual information. Through rigorous experiments on popular image datasets and computer vision tasks, our framework shows promising improvement compared to existing work, including superior performance in downstream tasks, better generalizability, ultra-high bandwidth efficiency, and low reconstruction latency.

TACO: Rethinking Semantic Communications with Task Adaptation and Context Embedding

TL;DR

This work tackles the challenge of semantic communications under evolving receiver goals by separating context information from task-specific content. It introduces TACO, a framework that embeds both context and GO content in a shared VQ-VAE latent space, uses GradCAM to guide ROI-based information selection, and leverages a Local Semantic Feedback loop for adaptive rate control and efficient latent fusion. The approach enables rapid adaptation to new downstream tasks with minimal additional bandwidth, improving reconstruction quality, task performance, and latency across image reconstruction, classification, and object detection benchmarks, while outperforming diffusion-based and conventional baselines. The findings suggest significant practical impact for bandwidth-efficient SemCom in dynamic multi-task settings and point to future work on channel noise and context-content compression techniques.

Abstract

Recent advancements in generative artificial intelligence have introduced groundbreaking approaches to innovating next-generation semantic communication, which prioritizes conveying the meaning of a message rather than merely transmitting raw data. A fundamental challenge in semantic communication lies in accurately identifying and extracting the most critical semantic information while adapting to downstream tasks without degrading performance, particularly when the objective at the receiver may evolve over time. To enable flexible adaptation to multiple tasks at the receiver, this work introduces a novel semantic communication framework, which is capable of jointly capturing task-specific information to enhance downstream task performance and contextual information. Through rigorous experiments on popular image datasets and computer vision tasks, our framework shows promising improvement compared to existing work, including superior performance in downstream tasks, better generalizability, ultra-high bandwidth efficiency, and low reconstruction latency.
Paper Structure (21 sections, 9 equations, 3 figures, 6 tables)

This paper contains 21 sections, 9 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. VQ-VAE
  4. Problem Formulation
  5. Method
  6. Design of the Transmitter
  7. Generation of the Image Context
  8. Identification of the Important Semantic Information
  9. Backbone System
  10. Local Semantic Feedback (LSF)
  11. Design of the Receiver
  12. Adaptation to New Downstream Tasks
  13. Experiments
  14. Scenario-1
  15. Image Reconstruction
  16. ...and 6 more sections

Figures (3)

  • Figure 1: The basic framework of TACO. The transmitter uses the encoder ($E$) to map the image to its latent space. Subsequently, the transmitter downsamples the image by a factor of 4 and utilizes the same $E$ to map the low-resolution image to the same latent space, which serves as the context representation. Meanwhile, the transmitter identifies the task-specific information using a neural network in the pixel domain. Then, the corresponding latent representation is extracted from the image latent. At the receiver side, the received context latent is first decoded to the pixel domain and is upsampled by a factor of 4 before being encoded back to the latent space. After deriving the latent, the receiver replaces the corresponding latent embeddings with the received latent indices representing task-oriented information. Finally, the same decoder is utilized to project the merged latent back to the pixel domain.
  • Figure 2: Local feedback for pre-defined tasks. The task-specific information extraction is one-time, but the information is presented partially as different percentages following [10, 20, 50, 100] percentages.
  • Figure 3: Object Detection for Task Switching in Different Steps