On nonlinear compression costs: when Shannon meets Rényi

TL;DR

This work analyzes nonlinear compression costs by minimizing the exponential average codeword length, which corresponds to Rényi entropy rather than Shannon entropy. It generalizes Arithmetic Coding to AC_q using escort distributions, showing analytically that the resulting exponential cost approaches the Rényi entropy $H_{1/(1+t)}[p]$ for i.i.d. sources, and demonstrates empirical gains on both simulated data and real Wikipedia text. The authors connect the exponential-cost objective to large-deviation theory via Cramér's theorem, justifying why minimizing $L(t)$ reduces the risk of long codewords and providing case-based insights into the optimal escort parameter $q$. They also discuss semi-static versus static distributions, estimation errors, and practical implications for threshold-bounded coding, delivering a comprehensive framework for risk-sensitive, exponential-cost compression with operational guidance and empirical validation.

Abstract

Shannon entropy is the shortest average codeword length a lossless compressor can achieve by encoding i.i.d. symbols. However, there are cases in which the objective is to minimize the \textit{exponential} average codeword length, i.e. when the cost of encoding/decoding scales exponentially with the length of codewords. The optimum is reached by all strategies that map each symbol $x_i$ generated with probability $p_i$ into a codeword of length $\ell^{(q)}_D(i)=-\log_D\frac{p_i^q}{\sum_{j=1}^Np_j^q}$. This leads to the minimum exponential average codeword length, which equals the Rényi, rather than Shannon, entropy of the source distribution. We generalize the established Arithmetic Coding (AC) compressor to this framework. We analytically show that our generalized algorithm provides an exponential average length which is arbitrarily close to the Rényi entropy, if the symbols to encode are i.i.d.. We then apply our algorithm to both simulated (i.i.d. generated) and real (a piece of Wikipedia text) datasets. While, as expected, we find that the application to i.i.d. data confirms our analytical results, we also find that, when applied to the real dataset (composed by highly correlated symbols), our algorithm is still able to significantly reduce the exponential average codeword length with respect to the classical `Shannonian' one. Moreover, we provide another justification of the use of the exponential average: namely, we show that by minimizing the exponential average length it is possible to minimize the probability that codewords exceed a certain threshold length. This relation relies on the connection between the exponential average and the cumulant generating function of the source distribution, which is in turn related to the probability of large deviations. We test and confirm our results again on both simulated and real datasets.

Figures (9)

• Figure 1: Exponential average codeword length of strings of length $M=50$, composed by i.i.d. symbols sampled according to $p_i\propto i^{-1}$, $i\in[1,3]$. Here $t=0.8$. Its minimum is reached in the proximity of the red vertical dashed line, corresponding to the optimal $q$, i.e. $q_t$. For that value of $q$, the distance with respect to the Rényi entropy of the string (flat orange line) is approximately $2$.
• Figure 2: Probability distribution of the individual symbols (i.e., characters) in the Wikipedia dataset. Symbols have been ordered by decreasing frequency and assigned a rank. The probability distribution has been estimated in a frequentist approach as $p_i=n_i/W$, with $n_i$ being the number of times that symbol $x_i$ appears in Wikipedia.
• Figure 3: This Figure shows, for the synthetic i.i.d. generated symbols, the empirical exponential average $L_M^{emp}(t,q)$ for $M=20$ (blue line), the Rényi entropy $M\cdot H_{q_t}[p]$ (horizontal orange line) with $q_t=1/(1+t)$ (dotted vertical red line). The three panels show different values of $t\in\{0.2,0.8,1.8\}$. We can see that, in each case, the minimum of $L_M^{emp}(t,q)$ is reached at $q=q_t$, and that $L_M^{emp}(t,q_t)- M\cdot H_{q_t}[p] \approx 2$.
• Figure 4: This Figure shows, for the real Wikipedia dataset,, the empirical exponential average $L_M^{emp}(t,q)$ for $M=20$ (blue line), the Rényi entropy $M\cdot H_{q_t}[p]$ (horizontal orange line) with $q_t=1/(1+t)$ (dotted vertical red line). The three panels show different values of $t\in\{0.2,0.8,1.8\}$. In all these cases, it is instead evident that the minimum of $L_M^{emp}(t,q)$ is reached for $q < q_t$. Additionally, $\min_q L_M^{emp}(q,t)$ could be lower (first panel) or almost exactly reach (second and third panels) the value $M\cdot H_{q_t}[p]$. We can also see that encoding according to AC$_{q_t}$ (i.e., see the intersection between the blue line and the dotted line) can lead to an exponential average length smaller than the Rényi entropy (first panel), or to an error which greater than $2$, i.e. $\min_q L_M^{emp}(t,q)-M\cdot H_{q_t}[p]>2$ (second and third panels).
• Figure 5: Empirical best $q$ for the Wikipedia dataset (blue line solid line) and $q_t$ (orange dashed line) for different values of $t$. For almost every value of $t$, the empirical optimal $q$ is smaller than $q_t$, meaning that in the Wikipedia dataset there is an abundance of 'rare' strings.