Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Convergence of Discrete Schemes for Computing the Rate-Distortion Function of Continuous Source

Lingyi Chen, Shitong Wu, Wenyi Zhang, Huihui Wu, Hao Wu

TL;DR

This paper establishes a rigorous link between discretized and continuous rate-distortion problems for continuous sources by embedding the problem in a sequence of finite-dimensional probability-measure spaces, and proves that discretized solutions converge to the continuous optimum with a rate $|f(\bm{r}^n)-f^*| \le C h$ where $h=2M/n^{1/d}$. It then derives algorithmic complexity bounds for the Blahut–Arimoto (BA) and Constrained BA (CBA) methods, showing $O\Big( \frac{m|\log\varepsilon|}{\varepsilon^{d+1}}\Big)$ and $O\Big( \frac{m|\log\varepsilon|}{\varepsilon^{d+1}}(1+\log|\log\varepsilon|)\Big)$ operations, respectively, to achieve $\varepsilon$-accuracy, with analogous results for the RD1 formulation. Numerical experiments on a uniform source corroborate the theory and illustrate convergence of the discrete RD solutions to the continuous optimum, and the framework is noted to extend to related problems like information bottleneck and RD-perception. The approach emphasizes that convergence is guaranteed by the discretization framework itself, not by any particular solver choice, enhancing the reliability of numerical RD computations for continuous sources.

Abstract

Computing the rate-distortion function for continuous sources is commonly regarded as a standard continuous optimization problem. When numerically addressing this problem, a typical approach involves discretizing the source space and subsequently solving the associated discrete problem. However, existing literature has predominantly concentrated on the convergence analysis of solving discrete problems, usually neglecting the convergence relationship between the original continuous optimization and its associated discrete counterpart. This neglect is not rigorous, since the solution of a discrete problem does not necessarily imply convergence to the solution of the original continuous problem, especially for non-linear problems. To address this gap, our study employs rigorous mathematical analysis, which constructs a series of finite-dimensional spaces approximating the infinite-dimensional space of the probability measure, establishing that solutions from discrete schemes converge to those from the continuous problems.

On Convergence of Discrete Schemes for Computing the Rate-Distortion Function of Continuous Source

TL;DR

This paper establishes a rigorous link between discretized and continuous rate-distortion problems for continuous sources by embedding the problem in a sequence of finite-dimensional probability-measure spaces, and proves that discretized solutions converge to the continuous optimum with a rate where . It then derives algorithmic complexity bounds for the Blahut–Arimoto (BA) and Constrained BA (CBA) methods, showing and operations, respectively, to achieve -accuracy, with analogous results for the RD1 formulation. Numerical experiments on a uniform source corroborate the theory and illustrate convergence of the discrete RD solutions to the continuous optimum, and the framework is noted to extend to related problems like information bottleneck and RD-perception. The approach emphasizes that convergence is guaranteed by the discretization framework itself, not by any particular solver choice, enhancing the reliability of numerical RD computations for continuous sources.

Abstract

Computing the rate-distortion function for continuous sources is commonly regarded as a standard continuous optimization problem. When numerically addressing this problem, a typical approach involves discretizing the source space and subsequently solving the associated discrete problem. However, existing literature has predominantly concentrated on the convergence analysis of solving discrete problems, usually neglecting the convergence relationship between the original continuous optimization and its associated discrete counterpart. This neglect is not rigorous, since the solution of a discrete problem does not necessarily imply convergence to the solution of the original continuous problem, especially for non-linear problems. To address this gap, our study employs rigorous mathematical analysis, which constructs a series of finite-dimensional spaces approximating the infinite-dimensional space of the probability measure, establishing that solutions from discrete schemes converge to those from the continuous problems.
Paper Structure (9 sections, 8 theorems, 112 equations, 2 figures)

This paper contains 9 sections, 8 theorems, 112 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. The Continuous RD Problem
  4. Discretization of the Continuous RD Problem
  5. Notations from Mathematical Analysis
  6. Convergence on Discrete Schemes
  7. Convergence Rate and Algorithm Complexity
  8. Numerical Experiments
  9. conclusion

Key Result

Theorem 1

Under Assumptions assump, the solutions $\bm{r}^n$ to the discrete problem RDd satisfy both value convergence and sequence convergence, i.e., where $f^*$ is the optimal value of the continuous problem RD.

Figures (2)

  • Figure 1: The discrete optimal reproduction produced by the BA algorithm for the slope $\beta=0.1$ (left) and $\beta=0.2$ (right).
  • Figure 2: The discrete optimal reproduction produced by the CBA algorithm for the distortion $D=4$ (left) and $D=3$ (right).

Theorems & Definitions (8)

  • Theorem 1
  • Theorem 2
  • Lemma 1
  • Theorem 3
  • Theorem 4
  • Theorem 5
  • Lemma 2
  • Theorem 6