Context-Aware Iterative Token Detection and Masked Transmission for Wireless Token Communication
Junyong Shin, Joohyuk Park, Jihong Park, Jinho Choi, Yo-Seb Jeon
TL;DR
This work addresses wireless transmission of discrete language tokens by exploiting contextual dependencies via a pretrained masked language model (MLM). It formulates a Bayesian, MAP-based framework where the receiver jointly uses MLM-derived contextual priors and channel observations to iteratively detect tokens, while the transmitter uses the same priors to mask highly predictable tokens for rate adaptation. The key contributions are (i) an iterative context-aware token detector at the receiver, (ii) a context-aware masking strategy at the transmitter that greedily masks low-entropy tokens to reduce transmission rate, and (iii) demonstrations on Europarl and WikiText-103 showing improved semantic fidelity (cosine similarity metrics) and effective rate adaptation under Rayleigh block-fading channels. The approach yields substantial improvements in reconstructed sentence quality and offers practical rate control for token-based wireless applications, with potential extensions to multimodal token streams.
Abstract
The success of large-scale language models has established tokens as compact and meaningful units for natural-language representation, which motivates token communication over wireless channels, where tokens are considered fundamental units for wireless transmission. We propose a context-aware token communication framework that uses a pretrained masked language model (MLM) as a shared contextual probability model between the transmitter (Tx) and receiver (Rx). At Rx, we develop an iterative token detection method that jointly exploits MLM-guided contextual priors and channel observations based on a Bayesian perspective. At Tx, we additionally introduce a context-aware masking strategy which skips highly predictable token transmission to reduce transmission rate. Simulation results demonstrate that the proposed framework substantially improves reconstructed sentence quality and supports effective rate adaptation under various channel conditions.