TokenCom: Vision-Language Model for Multimodal and Multitask Token Communications

TL;DR

TaiChi, a novel VLM framework designed for token communications, adopts a dual-visual tokenizer architecture that processes both high- and low-resolution images to collaboratively capture pixel-level details and global conceptual features, and is integrated into a multimodal and multitask token communication system equipped with a joint VLM-channel coding scheme.

Abstract

Visual-Language Models (VLMs), with their strong capabilities in image and text understanding, offer a solid foundation for intelligent communications. However, their effectiveness is constrained by limited token granularity, overlong visual token sequences, and inadequate cross-modal alignment. To overcome these challenges, we propose TaiChi, a novel VLM framework designed for token communications. TaiChi adopts a dual-visual tokenizer architecture that processes both high- and low-resolution images to collaboratively capture pixel-level details and global conceptual features. A Bilateral Attention Network (BAN) is introduced to intelligently fuse multi-scale visual tokens, thereby enhancing visual understanding and producing compact visual tokens. In addition, a Kolmogorov Arnold Network (KAN)-based modality projector with learnable activation functions is employed to achieve precise nonlinear alignment from visual features to the text semantic space, thus minimizing information loss. Finally, TaiChi is integrated into a multimodal and multitask token communication system equipped with a joint VLM-channel coding scheme. Experimental results validate the superior performance of TaiChi, as well as the feasibility and effectiveness of the TaiChi-driven token communication system.

Figures (12)

• Figure 1: The structure of the proposed TaiChi framework.
• Figure 2: Dual-visual tokenizer architecture.
• Figure 3: The computation pipeline of BAN.
• Figure 4: The structure of KAN.
• Figure 5: The structure of the TaiChi-driven token communication system.