On the Computability of Finding Capacity-Achieving Codes
Angelos Gkekas, Nikos A. Mitsiou, Ioannis Souldatos, George K. Karagiannidis
TL;DR
The paper investigates whether capacity-achieving codes for arbitrary discrete memoryless channels can be constructed algorithmically. It develops a constructive framework based on a $\mu$-recursive function FindCode and uses exhaustive search, encoded via Kleene's normal form, to output a block code with $R\geq$ target and maximum error $\lambda_{\max}<\varepsilon$ when $R<C(p_{Y|X})$ and the channel is computable. It extends the approach to computable real error bounds $\varepsilon$ (and discusses computable $R$), clarifying the necessary computability assumptions and showing a positive computability result under these weaker conditions. While feasible in principle, the method incurs exponential complexity; the work lays groundwork for future efficiency improvements and extensions to broader channel models such as finite-state channels.
Abstract
This work studies the problem of constructing capacity-achieving codes from an algorithmic perspective. Specifically, we prove that there exists a Turing machine which, given a discrete memoryless channel $p_{Y|X}$, a target rate $R$ less than the channel capacity $C(p_{Y|X})$, and an error tolerance $ε> 0$, outputs a block code $\mathcal{C}$ achieving a rate at least $R$ and a maximum block error probability below $ε$. The machine operates in the general case where all transition probabilities of $p_{Y|X}$ are computable real numbers, and the parameters $R$ and $ε$ are rational. The proof builds on Shannon's channel coding theorem and relies on an exhaustive search approach that systematically enumerates all codes of increasing block length until a valid code is found. This construction is formalized using the theory of recursive functions, yielding a $μ$-recursive function $\mathrm{FindCode} : \mathbb{N}^3 \rightharpoonup \mathbb{N}$ that takes as input appropriate encodings of $p_{Y|X}$, $R$, and $ε$, and, whenever $R < C(p_{Y|X})$, outputs an encoding of a valid code. By Kleene's normal form theorem, which establishes the computational equivalence between Turing machines and $μ$-recursive functions, we conclude that the problem is solvable by a Turing machine. This result can also be extended to the case where $ε$ is a computable real number, while we further discuss an analogous generalization of our analysis when $R$ is computable as well. We note that the assumptions that the probabilities of $p_{Y|X}$, as well as $ε$ and $R$, are computable real numbers cannot be further weakened, since computable reals constitute the largest subset of $\mathbb{R}$ representable by algorithmic means.