Knowledge-Base based Semantic Image Transmission Using CLIP
Chongyang Li, Yanmei He, Tianqian Zhang, Mingjian He, Shouyin Liu
TL;DR
The paper addresses semantic image transmission by shifting from pixel-level reconstruction to semantic-feature transmission and knowledge-base retrieval. It leverages CLIP to generate 512-d semantic embeddings, compresses them to a small dimensionality k with a lightweight encoder, transmits over Gaussian or Rayleigh channels, and uses FAISS to retrieve the closest KB embedding for the final image, formalized by the retrieval rule $\hat{x} = \arg\min_{kb_i \in KB} \|\hat{y}-kb_i\|_2$ and the channel model $\hat{z}=Hz+n$. Semantic accuracy is introduced as the evaluation metric, focusing on category-level correctness rather than traditional PSNR. Experiments on CIFAR100 show this approach outperforms traditional BPG+LDPC and SwinJSCC across various CBR and SNR conditions, with notable robustness and lower latency due to the lightweight encoder/decoder and FAISS-based retrieval. The work demonstrates a practical pathway toward semantic-aware communication with programmable bandwidth and real-time performance while highlighting future opportunities in KB optimization and more efficient feature compression.
Abstract
This paper proposes a novel knowledge-Base (KB) assisted semantic communication framework for image transmission. At the receiver, a Facebook AI Similarity Search (FAISS) based vector database is constructed by extracting semantic embeddings from images using the Contrastive Language-Image Pre-Training (CLIP) model. During transmission, the transmitter first extracts a 512-dimensional semantic feature using the CLIP model, then compresses it with a lightweight neural network for transmission. After receiving the signal, the receiver reconstructs the feature back to 512 dimensions and performs similarity matching from the KB to retrieve the most semantically similar image. Semantic transmission success is determined by category consistency between the transmitted and retrieved images, rather than traditional metrics like Peak Signal-to-Noise Ratio (PSNR). The proposed system prioritizes semantic accuracy, offering a new evaluation paradigm for semantic-aware communication systems. Experimental validation on CIFAR100 demonstrates the effectiveness of the framework in achieving semantic image transmission.