Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Computability of the Optimizer for Rate Distortion Functions

Jonathan E. W. Huffmann, Holger Boche

TL;DR

The paper investigates whether the test channel achieving the rate-distortion function can be computed. It shows that although the rate-distortion function $R(D)$ is computable for finite alphabets, the optimizer $P^*_{Y|X}$ is, in general, not Banach-Mazur (Turing) computable, even for simple distortion measures like the Hamming distortion. By constructing computable distortion sequences that encode undecidable sets, the authors prove non-computability of the optimizer and extend the result to fixed-distortion scenarios, implying there is no universal algorithm to compute the optimizer. Nonetheless, the rate-distortion function itself remains computable, highlighting a fundamental separation between the computability of $R(D)$ and its optimizer in lossy source coding.

Abstract

Rate distortion theory treats the problem of encoding a source with minimum codebook size while at the same time allowing for a certain amount of errors in the reconstruction measured by a fidelity criterion and distortion level. Similar to the channel coding problem the optimal rate of the codebook with respect to the blocklength is given by a convex optimization problem involving information theoretic quantities like mutual information. The value of the rate in dependence of the distortion level as well as the optimizer used in the codebook construction are of theoretical and practical importance in communication and information theory. In this paper the behavior of the rate distortion function regarding the computability of the optimizing test channel is investigated. We find that comparable with known results about the optimizer for other information theoretic problems a similar result is found to be true also regarding the computability of the optimizer for rate distortion functions. It turns out that while the rate distortion function is usually computable the optimizer for this problem is in general non-computable even for simple distortion measures.

Computability of the Optimizer for Rate Distortion Functions

TL;DR

The paper investigates whether the test channel achieving the rate-distortion function can be computed. It shows that although the rate-distortion function is computable for finite alphabets, the optimizer is, in general, not Banach-Mazur (Turing) computable, even for simple distortion measures like the Hamming distortion. By constructing computable distortion sequences that encode undecidable sets, the authors prove non-computability of the optimizer and extend the result to fixed-distortion scenarios, implying there is no universal algorithm to compute the optimizer. Nonetheless, the rate-distortion function itself remains computable, highlighting a fundamental separation between the computability of and its optimizer in lossy source coding.

Abstract

Rate distortion theory treats the problem of encoding a source with minimum codebook size while at the same time allowing for a certain amount of errors in the reconstruction measured by a fidelity criterion and distortion level. Similar to the channel coding problem the optimal rate of the codebook with respect to the blocklength is given by a convex optimization problem involving information theoretic quantities like mutual information. The value of the rate in dependence of the distortion level as well as the optimizer used in the codebook construction are of theoretical and practical importance in communication and information theory. In this paper the behavior of the rate distortion function regarding the computability of the optimizing test channel is investigated. We find that comparable with known results about the optimizer for other information theoretic problems a similar result is found to be true also regarding the computability of the optimizer for rate distortion functions. It turns out that while the rate distortion function is usually computable the optimizer for this problem is in general non-computable even for simple distortion measures.

Paper Structure

This paper contains 7 sections, 16 theorems, 129 equations, 7 figures.

Table of Contents

  1. Introduction
  2. Notation
  3. Prerequisites from Information Theory
  4. Analytic Calculation of the Rate Distortion Function
  5. Prerequisites from Computable Analysis
  6. Main Results and Discussion
  7. Conclusion

Key Result

Lemma 1

Let $P_{X}$ and $P_{Y}$ be probability distributions over the alphabet $\mathcal{X}=\mathcal{Y}$. Assume that $P_{X}\prec P_{Y}$ then we have $H(X)\geq H(Y)$.

Figures (7)

  • Figure 1: Turing machine (cf. Weihrauch2000)
  • Figure 2: Block diagram of a Turing machine which gets a representation of $A$ as input and computes a representation of $B$ as output.
  • Figure 3: Block diagram of a Turing machine computing $P^{*}_{Y|X}$. The Turing machine gets a representation for $P_{X}$, $d$ and $D$ as input and computes a representation of $P^{*}_{Y|X}$ as output.
  • Figure 4: Block diagram of a Turing machine computing a representation of $P^{*}_{Y|X}$ as output. Now the Turing machine only gets a representation of $d$ and $D$ as Input. Even if there exist a Turing machine of this type for every $P_{X}$ there does not necessarily exist a Turing machine of the more general kind shown in figure \ref{['fig:turing-picto1']}.
  • Figure 5: Erasure distortion measure with a second erasure symbol and $|\mathcal{X}|=2$, $|\mathcal{Y}|=4$
  • ...and 2 more figures

Theorems & Definitions (35)

  • Definition 1
  • Lemma 1
  • Definition 2
  • Theorem 1
  • Definition 3
  • Theorem 2
  • Definition 4
  • Definition 5
  • Lemma 2
  • Theorem 3
  • ...and 25 more