Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Constrained BA Algorithm for Rate-Distortion and Distortion-Rate Functions

Lingyi Chen, Shitong Wu, Wenhao Ye, Huihui Wu, Wenyi Zhang, Hao Wu, Bo Bai

TL;DR

This work introduces the Constrained Blahut-Arimoto (CBA) algorithm to directly compute the RD and DR functions for a given distortion by updating the Lagrange multiplier through a one-dimensional Newton root-finding step at each iteration. The authors prove an $O(1/n)$ convergence rate and a complexity of $O\left(\frac{MN\log N}{\varepsilon}(1+\log|\log \varepsilon|)\right)$ for achieving an $\varepsilon$-approximation, and demonstrate substantial empirical speedups over the classical BA, as well as AS and EM-based approaches, across discretized Gaussian, Laplacian, and uniform sources. The method also handles challenging RD-curve features such as bifurcations and phase transitions, preserving stability and accuracy while significantly reducing iteration counts. The framework’s generality suggests broad applicability to RD/DR problems and potential extensions to other constrained information-theoretic settings.

Abstract

The Blahut-Arimoto (BA) algorithm has played a fundamental role in the numerical computation of rate-distortion (RD) functions. This algorithm possesses a desirable monotonic convergence property by alternatively minimizing its Lagrangian with a fixed multiplier. In this paper, we propose a novel modification of the BA algorithm, wherein the multiplier is updated through a one-dimensional root-finding step using a monotonic univariate function, efficiently implemented by Newton's method in each iteration. Consequently, the modified algorithm directly computes the RD function for a given target distortion, without exploring the entire RD curve as in the original BA algorithm. Moreover, this modification presents a versatile framework, applicable to a wide range of problems, including the computation of distortion-rate (DR) functions. Theoretical analysis shows that the outputs of the modified algorithms still converge to the solutions of the RD and DR functions with rate $O(1/n)$, where $n$ is the number of iterations. Additionally, these algorithms provide $\varepsilon$-approximation solutions with $O\left(\frac{MN\log N}{\varepsilon}(1+\log |\log \varepsilon|)\right)$ arithmetic operations, where $M,N$ are the sizes of source and reproduced alphabets respectively. Numerical experiments demonstrate that the modified algorithms exhibit significant acceleration compared with the original BA algorithms and showcase commendable performance across classical source distributions such as discretized Gaussian, Laplacian and uniform sources.

A Constrained BA Algorithm for Rate-Distortion and Distortion-Rate Functions

TL;DR

This work introduces the Constrained Blahut-Arimoto (CBA) algorithm to directly compute the RD and DR functions for a given distortion by updating the Lagrange multiplier through a one-dimensional Newton root-finding step at each iteration. The authors prove an convergence rate and a complexity of for achieving an -approximation, and demonstrate substantial empirical speedups over the classical BA, as well as AS and EM-based approaches, across discretized Gaussian, Laplacian, and uniform sources. The method also handles challenging RD-curve features such as bifurcations and phase transitions, preserving stability and accuracy while significantly reducing iteration counts. The framework’s generality suggests broad applicability to RD/DR problems and potential extensions to other constrained information-theoretic settings.

Abstract

The Blahut-Arimoto (BA) algorithm has played a fundamental role in the numerical computation of rate-distortion (RD) functions. This algorithm possesses a desirable monotonic convergence property by alternatively minimizing its Lagrangian with a fixed multiplier. In this paper, we propose a novel modification of the BA algorithm, wherein the multiplier is updated through a one-dimensional root-finding step using a monotonic univariate function, efficiently implemented by Newton's method in each iteration. Consequently, the modified algorithm directly computes the RD function for a given target distortion, without exploring the entire RD curve as in the original BA algorithm. Moreover, this modification presents a versatile framework, applicable to a wide range of problems, including the computation of distortion-rate (DR) functions. Theoretical analysis shows that the outputs of the modified algorithms still converge to the solutions of the RD and DR functions with rate , where is the number of iterations. Additionally, these algorithms provide -approximation solutions with arithmetic operations, where are the sizes of source and reproduced alphabets respectively. Numerical experiments demonstrate that the modified algorithms exhibit significant acceleration compared with the original BA algorithms and showcase commendable performance across classical source distributions such as discretized Gaussian, Laplacian and uniform sources.
Paper Structure (12 sections, 7 theorems, 71 equations, 3 figures, 3 tables, 2 algorithms)

This paper contains 12 sections, 7 theorems, 71 equations, 3 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Constrained BA Algorithm
  3. Constrained BA algorithm for the RD function
  4. Constrained BA algorithm for DR function
  5. Convergence Analysis
  6. Convergence analysis for the RD function
  7. Convergence analysis for the DR function
  8. Numerical Results and Discussions
  9. Accuracy and Efficiency
  10. Convergence Behaviour Near Bifurcations
  11. Phase Transitions in the Uniform Source
  12. Conclusion

Key Result

Theorem 3.1

Denoting by $(\boldsymbol{w}^{(n)},\boldsymbol{r}^{(n)})$ the sequence generated by the CBA algorithm and by $(\boldsymbol{w}^{*},\boldsymbol{r}^{*})$ the global optimizer of RD function semi_CommOT_model, we have i.e., the CBA algorithm converges to the RD function with rate $O(1/n)$.

Figures (3)

  • Figure 1: Comparison between convergent trajectories of the CBA algorithm and the BA algorithm, for computing $R(D)$ with $D = 0.5$.
  • Figure 2: The RD curve produced by the CBA algorithm and the BA algorithm (Left), and comparison between the numbers of iterations (Right). Here, the stop condition is set as that the decrease from the previous step is below $10^{-8}$. For each given target distortion $D$, the BA algorithm is executed using the slope $\lambda$ already computed by the CBA algorithm with high accuracy as the predetermined parameter.
  • Figure 3: The discrete optimal reproduction produced by the CBA algorithm for the cases of distortion $D=4$ (Left) and $D=8$ (Right).

Theorems & Definitions (10)

  • Remark 2.1
  • Theorem 3.1
  • Theorem 3.2
  • Remark 3.1
  • Theorem 3.3
  • Theorem 3.4
  • Lemma 3.1
  • Lemma 3.2
  • Lemma 3.3
  • Remark 3.2