Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Visual Fidelity Index for Generative Semantic Communications with Critical Information Embedding

Jianhao Huang, Qunsong Zeng, Kaibin Huang

TL;DR

This work addresses the inefficiency and fidelity gap in purely prompt-driven Gen-SemCom by proposing a hybrid framework that embeds semantically critical pixel information alongside text prompts. A critical information embedding (CIE) pipeline uses CAM-based semantic filtering to select salient features and a diffusion-based generator to reconstruct high-fidelity images from prompts and critical pixels. The authors introduce the generative visual information fidelity (GVIF) metric, grounded in GSM feature statistics and a human-visual-system (HVS) model, to quantify how much perceptual information is preserved after transmission and generation. They then formulate a channel-adaptive optimization that maximizes GVIF under latency and distortion constraints, and validate the approach with experiments on ImageNet, showing improved PSNR in critical regions and lower FID compared to benchmarks, with GVIF correlating to perceptual quality and robustness to channel conditions.

Abstract

Generative semantic communication (Gen-SemCom) with large artificial intelligence (AI) model promises a transformative paradigm for 6G networks, which reduces communication costs by transmitting low-dimensional prompts rather than raw data. However, purely prompt-driven generation loses fine-grained visual details. Additionally, there is a lack of systematic metrics to evaluate the performance of Gen-SemCom systems. To address these issues, we develop a hybrid Gen-SemCom system with a critical information embedding (CIE) framework, where both text prompts and semantically critical features are extracted for transmissions. First, a novel approach of semantic filtering is proposed to select and transmit the semantically critical features of images relevant to semantic label. By integrating the text prompt and critical features, the receiver reconstructs high-fidelity images using a diffusion-based generative model. Next, we propose the generative visual information fidelity (GVIF) metric to evaluate the visual quality of the generated image. By characterizing the statistical models of image features, the GVIF metric quantifies the mutual information between the distorted features and their original counterparts. By maximizing the GVIF metric, we design a channel-adaptive Gen-SemCom system that adaptively control the volume of features and compression rate according to the channel state. Experimental results validate the GVIF metric's sensitivity to visual fidelity, correlating with both the PSNR and critical information volume. In addition, the optimized system achieves superior performance over benchmarking schemes in terms of higher PSNR and lower FID scores.

Visual Fidelity Index for Generative Semantic Communications with Critical Information Embedding

TL;DR

This work addresses the inefficiency and fidelity gap in purely prompt-driven Gen-SemCom by proposing a hybrid framework that embeds semantically critical pixel information alongside text prompts. A critical information embedding (CIE) pipeline uses CAM-based semantic filtering to select salient features and a diffusion-based generator to reconstruct high-fidelity images from prompts and critical pixels. The authors introduce the generative visual information fidelity (GVIF) metric, grounded in GSM feature statistics and a human-visual-system (HVS) model, to quantify how much perceptual information is preserved after transmission and generation. They then formulate a channel-adaptive optimization that maximizes GVIF under latency and distortion constraints, and validate the approach with experiments on ImageNet, showing improved PSNR in critical regions and lower FID compared to benchmarks, with GVIF correlating to perceptual quality and robustness to channel conditions.

Abstract

Generative semantic communication (Gen-SemCom) with large artificial intelligence (AI) model promises a transformative paradigm for 6G networks, which reduces communication costs by transmitting low-dimensional prompts rather than raw data. However, purely prompt-driven generation loses fine-grained visual details. Additionally, there is a lack of systematic metrics to evaluate the performance of Gen-SemCom systems. To address these issues, we develop a hybrid Gen-SemCom system with a critical information embedding (CIE) framework, where both text prompts and semantically critical features are extracted for transmissions. First, a novel approach of semantic filtering is proposed to select and transmit the semantically critical features of images relevant to semantic label. By integrating the text prompt and critical features, the receiver reconstructs high-fidelity images using a diffusion-based generative model. Next, we propose the generative visual information fidelity (GVIF) metric to evaluate the visual quality of the generated image. By characterizing the statistical models of image features, the GVIF metric quantifies the mutual information between the distorted features and their original counterparts. By maximizing the GVIF metric, we design a channel-adaptive Gen-SemCom system that adaptively control the volume of features and compression rate according to the channel state. Experimental results validate the GVIF metric's sensitivity to visual fidelity, correlating with both the PSNR and critical information volume. In addition, the optimized system achieves superior performance over benchmarking schemes in terms of higher PSNR and lower FID scores.
Paper Structure (34 sections, 1 theorem, 26 equations, 11 figures, 1 algorithm)

This paper contains 34 sections, 1 theorem, 26 equations, 11 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Hybrid Gen-SemCom System
  3. Critical Information Embedding
  4. Semantic-level prompts
  5. Critical pixel information
  6. System Operations and Models
  7. Critical information extraction
  8. Transmission model
  9. GenAI-enabled recovery
  10. Diffusion-based denoiser
  11. Training of Source Coder
  12. Semantic Filtering
  13. Semantic Importance Modeling
  14. Feature Filtering
  15. Side Information Transmission
  16. ...and 19 more sections

Key Result

Proposition 4.1

For an image sample $\bm{x}$ with the realizations $\bar{\bm{\theta}}^{r}$ and $\bar{\bm{\beta}}$, and given the variance of visual noise ${\gamma}^2$ and the filtering set $\mathbb{P}$, the GVIF can be expressed by where $\bar{\theta}_{ijc}^{r}$ and $\bar{\beta}_{ijc}$ are the $(i,j,c)$-th element of tensor parameter $\bar{\bm{\theta}}^{r}$ and $\bar{\bm{\beta}}$, respectively, and $\mathbb{U}=\

Figures (11)

  • Figure 1: Comparisons of multi-level SemCom systems with different visual metrics.
  • Figure 2: The hybrid Gen-SemCom system with the CIE process.
  • Figure 3: The system operations of the CIE process. Conv, $\left\lceil \cdot \right\rfloor$, EE, and ED represent the convolutional layer, scalar quantization, entropy encoder, and entropy decoder, respectively.
  • Figure 5: Computation procedures of the GVIF metric.
  • Figure 6: Illustrations of the GVIF's ability for image content understanding. In this image example, we actively choose different set $\mathbb{P}$ to select the image features. For better illustrations, we upsample the sets into image size as regions. We set $\bar{\beta}_{ijc}=1, \forall i,j,c$ and $\gamma=0.1$. The presented mutual information is averaged over feature channels.
  • ...and 6 more figures

Theorems & Definitions (2)

  • Remark 3.1
  • Proposition 4.1