Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Communication-Efficient Hybrid Language Model via Uncertainty-Aware Opportunistic and Compressed Transmission

Seungeun Oh, Jinhyuk Kim, Jihong Park, Seung-Woo Ko, Jinho Choi, Tony Q. S. Quek, Seong-Lyun Kim

TL;DR

CU-HLM tackles the bottleneck of token throughput in wireless-edge hybrid language models by exploiting a strong link between on-device uncertainty and LLM rejection. It combines uncertainty-aware opportunistic transmission to skip costly uplinks and LLM runs with uncertainty-aware vocabulary compression to reduce remaining uplink payload, supported by theoretical thresholds and distortion bounds. Empirical results show up to 206× token throughput gains with minimal accuracy loss (around 97% of full LLM accuracy) across diverse channels, datasets, and model sizes, with online compression achieving near-offline performance without server-side feedback. The framework offers a practical path to scalable, bandwidth-efficient on-device language inference in bandwidth-constrained edge environments.

Abstract

To support emerging language-based applications using dispersed and heterogeneous computing resources, the hybrid language model (HLM) offers a promising architecture, where an on-device small language model (SLM) generates draft tokens that are validated and corrected by a remote large language model (LLM). However, the original HLM suffers from substantial communication overhead, as the LLM requires the SLM to upload the full vocabulary distribution for each token. Moreover, both communication and computation resources are wasted when the LLM validates tokens that are highly likely to be accepted. To overcome these limitations, we propose communication-efficient and uncertainty-aware HLM (CU-HLM). In CU-HLM, the SLM transmits truncated vocabulary distributions only when its output uncertainty is high. We validate the feasibility of this opportunistic transmission by discovering a strong correlation between SLM's uncertainty and LLM's rejection probability. Furthermore, we theoretically derive optimal uncertainty thresholds and optimal vocabulary truncation strategies. Simulation results show that, compared to standard HLM, CU-HLM achieves up to 206$\times$ higher token throughput by skipping 74.8% transmissions with 97.4% vocabulary compression, while maintaining 97.4% accuracy.

Communication-Efficient Hybrid Language Model via Uncertainty-Aware Opportunistic and Compressed Transmission

TL;DR

CU-HLM tackles the bottleneck of token throughput in wireless-edge hybrid language models by exploiting a strong link between on-device uncertainty and LLM rejection. It combines uncertainty-aware opportunistic transmission to skip costly uplinks and LLM runs with uncertainty-aware vocabulary compression to reduce remaining uplink payload, supported by theoretical thresholds and distortion bounds. Empirical results show up to 206× token throughput gains with minimal accuracy loss (around 97% of full LLM accuracy) across diverse channels, datasets, and model sizes, with online compression achieving near-offline performance without server-side feedback. The framework offers a practical path to scalable, bandwidth-efficient on-device language inference in bandwidth-constrained edge environments.

Abstract

To support emerging language-based applications using dispersed and heterogeneous computing resources, the hybrid language model (HLM) offers a promising architecture, where an on-device small language model (SLM) generates draft tokens that are validated and corrected by a remote large language model (LLM). However, the original HLM suffers from substantial communication overhead, as the LLM requires the SLM to upload the full vocabulary distribution for each token. Moreover, both communication and computation resources are wasted when the LLM validates tokens that are highly likely to be accepted. To overcome these limitations, we propose communication-efficient and uncertainty-aware HLM (CU-HLM). In CU-HLM, the SLM transmits truncated vocabulary distributions only when its output uncertainty is high. We validate the feasibility of this opportunistic transmission by discovering a strong correlation between SLM's uncertainty and LLM's rejection probability. Furthermore, we theoretically derive optimal uncertainty thresholds and optimal vocabulary truncation strategies. Simulation results show that, compared to standard HLM, CU-HLM achieves up to 206 higher token throughput by skipping 74.8% transmissions with 97.4% vocabulary compression, while maintaining 97.4% accuracy.
Paper Structure (24 sections, 3 theorems, 35 equations, 13 figures, 2 tables, 1 algorithm)

This paper contains 24 sections, 3 theorems, 35 equations, 13 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Contributions and Paper Organization
  4. System Model
  5. Token Generation of a Hybrid Language Model
  6. Wireless Communication
  7. Token Throughput
  8. Uncertainty-Aware Opportunistic Hybrid Language Model
  9. Uncertainty and Rejection Prediction
  10. Uncertainty-Aware Opportunistic Transmission.
  11. Uncertainty Threshold
  12. Compressed Vocabulary Transmission for Communication-Efficient U-HLM
  13. Vocabulary Compression
  14. Compressed Vocabulary Transmission
  15. Offline Vocabulary Compression
  16. ...and 9 more sections

Key Result

Theorem 1

Assuming i.i.d. uncertainty $u(t)$ and rejection probability $\beta_d(t)$, let $u \coloneqq u(t)$ and $\beta \coloneqq \beta_d(t)$ for any $t$. Besides, we regard $\beta$ equivalent to its expectation $\mathbb{E}[\beta]$ of eq:10. Then, the uncertainty threshold $u_{\text{th}}$ is given as the upper where $\Delta = P(y_d < x_d)$ denotes the probability that a token is not deterministically accepte

Figures (13)

  • Figure 1: Comparison of inference accuracy and token throughput, where token throughput is inversely proportional to end-to-end latency, including communication and computation delays (SNR 10 dB, Rayleigh fading).
  • Figure 2: Uncertainty vs. rejection probability. Solid lines show mean; shaded regions denote 95% CI. Correlation coefficients: temperature perturbation (0.71), MC Dropout (0.65), and prompt perturbation (0.20).
  • Figure 3: Schematic illustration of U-HLM framework.
  • Figure 4: Linear relationship between uncertainty and rejection probability. Dashed vertical lines indicate the theoretical risk-averse and risk-prone thresholds, respectively.
  • Figure 5: Probability Density Function (PDF) of uncertainty, obtained via Gaussian kernel density estimation (KDE).
  • ...and 8 more figures

Theorems & Definitions (9)

  • Remark 1: Linear Correlation Between Uncertainty and Rejection Probability
  • Theorem 1: Uncertainty Threshold and Rejection Risk
  • proof
  • Proposition 1: Upper Bound on Total Variation Distance
  • proof
  • Remark 2: Offline Vocabulary Compression
  • Proposition 2: Approximated Upper Bound on $\mathrm{U}_{\text{TV}}(k,t)$
  • proof
  • Remark 3: Uncertainty-Aware Online Vocabulary Compression