Task-Oriented Multimodal Token Transmission in Resource-Constrained Multiuser Networks

TL;DR

This work tackles the bandwidth, power, and latency burden of transmitting transformer-generated multimodal tokens in resource-constrained networks by introducing a task-oriented framework that fuses multimodal information at the edge. A two-stage training approach—cross-modal alignment followed by task-oriented fine-tuning with a LoRA adapter—enables robust cross-modal fusion and task performance using fewer tokens. The transmission framework employs sliding-window pooling to compress non-text tokens and an OFDM-based receiver, while an alternating optimization scheme jointly optimizes bandwidth, power, and token length to balance latency and validation loss. Results demonstrate improved accuracy and reduced total cost across varying SNRs and budgets, highlighting the practical viability of efficient multimodal token transmission for edge-augmented foundation-model tasks.

Abstract

With the emergence of large model-based agents, widely adopted transformer-based architectures inevitably produce excessively long token embeddings for transmission, which may result in high bandwidth overhead, increased power consumption and latency. In this letter, we propose a task-oriented multimodal token transmission scheme for efficient multimodal information fusion and utilization. To improve the efficiency of token transmission, we design a two-stage training algotithm, including cross-modal alignment and task-oriented fine-tuning, for large model-based token communication. Meanwhile, token compression is performed using a sliding window pooling operation to save communication resources. To balance the trade-off between latency and model performance caused by compression, we formulate a weighted-sum optimization problem over latency and validation loss. We jointly optimizes bandwidth, power allocation, and token length across users by using an alternating optimization method. Simulation results demonstrate that the proposed algorithm outperforms the baseline under different bandwidth and power budgets. Moreover, the two-stage training algorithm achieves higher accuracy across various signal-to-noise ratios than the method without cross-modal alignment.

Figures (5)

• Figure 1: Task-oriented multimodal token transmission in resource-constrained multiuser networks.
• Figure 2: Total cost vs. $B^{\rm max}$
• Figure 3: Total cost vs. $P^{\rm max}$
• Figure 4: Impact of $\lambda$
• Figure 5: Accuracy vs. SNR