Log-Likelihood Loss for Semantic Compression

TL;DR

This work introduces a log-likelihood distortion $d_{\ell\ell}(x,y) = -\log P_{X|U}(x|y)$ to capture semantic fidelity in compression, defining the corresponding RDF $R_{\ell\ell}(D)$ over reconstructions $Y$ drawn from a fixed conditional model $P_{X|U}$. It establishes connections to standard log-loss RD, demonstrates a general single-parameter dual representation under a condition that yields a tight Shannon-type bound, and provides concrete binary and Gaussian examples to illustrate the framework. The paper also links this approach to rate-distortion-perception, showing how a CP structure on the distortion matrix enables achieving RD with perfect perception via an auxiliary variable $Z$, thereby unifying semantic fidelity, probabilistic reconstruction, and perceptual constraints. Overall, the results place semantic-oriented compression within a rigorous rate-distortion theory that naturally integrates generative decoding and perception constraints.

Abstract

We study lossy source coding under a distortion measure defined by the negative log-likelihood induced by a prescribed conditional distribution $P_{X|U}$. This \emph{log-likelihood distortion} models compression settings in which the reconstruction is a semantic representation from which the source can be probabilistically generated, rather than a pointwise approximation. We formulate the corresponding rate-distortion problem and characterize fundamental properties of the resulting rate-distortion function, including its connections to lossy compression under log-loss, classical rate-distortion problems with arbitrary distortion measures, and rate-distortion with perfect perception.

Figures (4)

• Figure 1: Log-likelihood loss based lossy compression.
• Figure 2:
• Figure 3: $(X,P_{X|U})$: $X \sim \mathrm{Ber}(0.25)$ and $P_{X|U} \sim \mathrm{BSC}(0.1)$, the figure plots $R_{\ell}(D)$ for $D \in [0,H(X)=0.8113]$ and $R_{\ell\mkern-2mu\ell}(D)$ for $D \in [D_{\min} =0.152, D_{\max}=0.945]$. At $D^{*}= 0.469$, both RDFs coincide. $D^{*}$ is the special operating point for $(X,P_{X|U})$ with $U \sim \mathrm{Ber}(0.1875)$.
• Figure 4: $(X,P_{X|U})$: $X \sim \mathrm{Ber}(0.35)$ and $P_{X|U} = [ 0.8, 0.4, 0.2; 0.2, 0.6, 0.8]$, the figure plots $R_{\ell}(D)$ for $D \in [0,0.934]$ and $R_{\ell\mkern-2mu\ell}(D)$ for $D \in [0.322, 1.022]$. Both RDFs coincide for $D^{*} \in [0.718,0.816]$. Every $D^{*}$ in this interval is a special operating point with a corresponding $(U_i,X)$ consistent with $(X,P_{X|U})$.Since RLLDF $R_{\ell\mkern-2mu\ell}(D)$ does not exhibits a closed-form expression for this $(X,P_{X|U})$, the $R_{\ell\mkern-2mu\ell}(D)$ is plotted using Blahut-Arimoto algorithm blahutarimoto.

Theorems & Definitions (19)

• Definition 1: Rate-Distortion Function (RDF)
• Definition 2: Rate-Distortion Function under Log-Likelihood Loss (RLLDF)
• Remark 1
• Theorem 1
• proof
• Theorem 2
• proof
• Corollary 1
• Theorem 3
• proof