Discrete Layered Entropy, Conditional Compression and a Tighter Strong Functional Representation Lemma

TL;DR

The paper introduces the discrete layered entropy Λ(p) as a piecewise-linear, computable surrogate for Shannon entropy with a crucial conditioning property Λ(X|Y)=Λ(X\ Y). It shows Λ underpins conditional encoding, linear programming formulations for maximum-entropy problems, and tight bounds for one-shot channel simulation and the strong functional representation lemma, improving upon prior bounds by exploiting Λ’s properties. It also connects Λ to non-prefix (one-to-one) codes, develops the discrete m-layered entropy Λ_[m](X) for arbitrarily close H(X) approximations via linear programs, and extends the framework to Rényi layers Λ_α(p). The work provides a cohesive theory that blends coding, mixture distributions, and channel simulation, yielding practically relevant bounds and LP-friendly approximations with near-optimal guarantees. These tools enhance both theoretical understanding and algorithmic approaches to entropy-related problems in information theory and related applications where linear programming and one-shot settings are central.

Abstract

We study a quantity called discrete layered entropy, which approximates the Shannon entropy within a logarithmic gap. Compared to the Shannon entropy, the discrete layered entropy is piecewise linear, approximates the expected length of the optimal one-to-one non-prefix code, and satisfies an elegant conditioning property. These properties make it useful for approximating the Shannon entropy in linear programming and maximum entropy problems, studying the optimal length of conditional encoding, and bounding the entropy of monotonic mixture distributions. In particular, it can give a bound $I(X;Y)+\log(I(X;Y)+3.4)+1$ for the strong functional representation lemma which is optimal within $2.8$ bits, and significantly improves upon the best known bound.

Figures (8)

• Figure 1: Left: Contour plot of the Shannon entropy $H(p)$ for $p:\{1,2,3\}\to[0,1]$ being a ternary probability mass function. Right: Contour plot of the discrete layered entropy $\Lambda(p)$. We can see that $\Lambda(p)$ is piecewise linear and lower-bounds $H(p)$. The red points are the points where $\Lambda(p)=H(p)$. They are also the vertices of the polytope $\{(p,z)\in\mathbb{R}^{3}\times\mathbb{R}:\,0\le z\le\Lambda(p)\}$.
• Figure 2: Top: The conditional encoding setting. Bottom: The one-shot channel simulation setting.
• Figure 3: Top: Diagram of the conditional variable-length encoding setting, where the encoder writes a variable-length description $M$ followed by a stream of other bits transmitted to the decoder. The side information $Y$ and the description length $|M|$ are optionally given to the decoder. Bottom: Whether the decoder can recover $X$ and/or $|M|$ (i.e., keep synchronized) with or without being given $Y,|M|$, under the three settings (e.g., for conditional prefix codes, if $Y$ is available, the decoder can recover $X,|M|$).
• Figure 4: Illustration of $\mathcal{R}(X)=\bigcup_{p_{Y|X}}\{(H(X|Y),H(X\backslash Y))\}$ showing the extreme points $(\Lambda(X),H(X))$ and $(H(X),H(X))$.
• Figure 5: Top: Diagram of one-shot variable-length channel simulation, where the encoder writes a variable-length description $M$ followed by a stream of other bits transmitted to the decoder. The common randomness $S$ and the description length $|M|$ are optionally given to the decoder. Bottom: Whether the decoder can output $Y$ and/or $|M|$ (i.e., keep synchronized) with or without being given $S,|M|$, under the three settings (e.g., for conditional prefix codes, if $S$ is available, the decoder can output $Y,|M|$).

Theorems & Definitions (31)

• Definition 1: Discrete layered entropy
• Proposition 2: Alternative definitions
• Proposition 3: Basic properties
• Remark 4
• Proposition 5: $\Lambda(X)\approx H(X)$
• Proposition 6
• Definition 7: Conditional compression
• Proposition 8
• Theorem 9: Alternative definition of $\Lambda(X)$
• Proposition 10: Conditioning property