Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

In-Context Source and Channel Coding

Ziqiong Wang, Tianqi Ren, Rongpeng Li, Zhifeng Zhao, Honggang Zhang

TL;DR

SSCC suffers a cliff effect at low SNR, where residual channel errors catastrophically disrupt LLM-driven arithmetic decoding. The authors propose receiver-side In-Context Decoding (ICD), coupling ECCT-based bit-wise reliability with contextual information through a three-stage pipeline: In-Context Candidate Generator (CCG), In-Context Candidate Sampler (CCS), and In-Context Likelihood Ranking (CLR), finalized by a fusion of reliability and linguistic likelihood. They provide stability and convergence guarantees for CCS and demonstrate through extensive AWGN and Rayleigh-channel experiments that ICD consistently outperforms conventional SSCC baselines and representative JSCC schemes, with favorable complexity thanks to diversity-aware sampling. ICD scales with larger LLM backbones and preserves transmitter-agnostic operation, offering a practical path to more robust text transmission under challenging channel conditions.

Abstract

Separate Source-Channel Coding (SSCC) remains attractive for text transmission due to its modularity and compatibility with mature entropy coders and powerful channel codes. However, SSCC often suffers from a pronounced cliff effect in low Signal-to-Noise Ratio (SNR) regimes, where residual bit errors after channel decoding can catastrophically break lossless source decoding, especially for Arithmetic Coding (AC) driven by Large Language Models (LLMs). This paper proposes a receiver-side In-Context Decoding (ICD) framework that enhances SSCC robustness without modifying the transmitter. ICD leverages an Error Correction Code Transformer (ECCT) to obtain bit-wise reliability for the decoded information bits. Based on the context-consistent bitstream, ICD constructs a confidence-ranked candidate pool via reliability-guided bit flipping, samples a compact yet diverse subset of candidates, and applies an LLM-based arithmetic decoder to obtain both reconstructions and sequence-level log-likelihoods. A reliability-likelihood fusion rule then selects the final output. We further provide theoretical guarantees on the stability and convergence of the proposed sampling procedure. Extensive experiments over Additive White Gaussian Noise (AWGN) and Rayleigh fading channels demonstrate consistent gains compared with conventional SSCC baselines and representative Joint Source-Channel Coding (JSCC) schemes.

In-Context Source and Channel Coding

TL;DR

SSCC suffers a cliff effect at low SNR, where residual channel errors catastrophically disrupt LLM-driven arithmetic decoding. The authors propose receiver-side In-Context Decoding (ICD), coupling ECCT-based bit-wise reliability with contextual information through a three-stage pipeline: In-Context Candidate Generator (CCG), In-Context Candidate Sampler (CCS), and In-Context Likelihood Ranking (CLR), finalized by a fusion of reliability and linguistic likelihood. They provide stability and convergence guarantees for CCS and demonstrate through extensive AWGN and Rayleigh-channel experiments that ICD consistently outperforms conventional SSCC baselines and representative JSCC schemes, with favorable complexity thanks to diversity-aware sampling. ICD scales with larger LLM backbones and preserves transmitter-agnostic operation, offering a practical path to more robust text transmission under challenging channel conditions.

Abstract

Separate Source-Channel Coding (SSCC) remains attractive for text transmission due to its modularity and compatibility with mature entropy coders and powerful channel codes. However, SSCC often suffers from a pronounced cliff effect in low Signal-to-Noise Ratio (SNR) regimes, where residual bit errors after channel decoding can catastrophically break lossless source decoding, especially for Arithmetic Coding (AC) driven by Large Language Models (LLMs). This paper proposes a receiver-side In-Context Decoding (ICD) framework that enhances SSCC robustness without modifying the transmitter. ICD leverages an Error Correction Code Transformer (ECCT) to obtain bit-wise reliability for the decoded information bits. Based on the context-consistent bitstream, ICD constructs a confidence-ranked candidate pool via reliability-guided bit flipping, samples a compact yet diverse subset of candidates, and applies an LLM-based arithmetic decoder to obtain both reconstructions and sequence-level log-likelihoods. A reliability-likelihood fusion rule then selects the final output. We further provide theoretical guarantees on the stability and convergence of the proposed sampling procedure. Extensive experiments over Additive White Gaussian Noise (AWGN) and Rayleigh fading channels demonstrate consistent gains compared with conventional SSCC baselines and representative Joint Source-Channel Coding (JSCC) schemes.
Paper Structure (18 sections, 3 theorems, 34 equations, 8 figures, 5 tables, 1 algorithm)

This paper contains 18 sections, 3 theorems, 34 equations, 8 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. LLM-based Source Coding
  4. Error Correction Code Transformer
  5. System Model and Problem Formulation
  6. System Model
  7. Problem Formulation
  8. The Framework of ICD
  9. In-Context Candidate Generator
  10. In-Context Candidate Sampler
  11. In-Context Likelihood Ranking
  12. Simulation Settings and Results
  13. Simulation Settings
  14. Overall System-Level Performance Comparison
  15. Impact of Confidence-Ranked and Sampled Candidates Hyperparameters
  16. ...and 3 more sections

Key Result

Theorem 1

Consider the Markov chain $\{\mathcal{S}^{(t)}\}_{t\ge 0}$ induced by the CCS sampler on the state space $\Omega$, it satisfies the following properties: (1) The state space $\Omega$ is finite. (2) The transition kernel $P$ meets the detailed balance condition with respect to the target distribution

Figures (8)

  • Figure 1: Framework of the Proposed ICD-Aided SSCC System.
  • Figure 2: Framework of ICD.
  • Figure 3: Distribution of correct and erroneous bits across confidence levels.
  • Figure 4: Overall system-level performance under AWGN channels.
  • Figure 5: Overall system-level performance under Rayleigh channels.
  • ...and 3 more figures

Theorems & Definitions (6)

  • Definition 1: Stationarity under Detailed Balance levin2017markov
  • Theorem 1: Properties of CCS
  • proof
  • Lemma 1: Ergodicity and Convergence Norris1998MarkovChain
  • Theorem 2: Stationarity and Convergence of CCS
  • proof