Minimum Entropy Coupling with Bottleneck

TL;DR

This paper investigates a novel lossy compression framework operating under logarithmic loss, designed to handle situations where the reconstruction distribution diverges from the source distribution, and proposes a greedy algorithm for EBIM with guaranteed performance and characterize the optimal solution near functional mappings.

Abstract

This paper investigates a novel lossy compression framework operating under logarithmic loss, designed to handle situations where the reconstruction distribution diverges from the source distribution. This framework is especially relevant for applications that require joint compression and retrieval, and in scenarios involving distributional shifts due to processing. We show that the proposed formulation extends the classical minimum entropy coupling framework by integrating a bottleneck, allowing for a controlled degree of stochasticity in the coupling. We explore the decomposition of the Minimum Entropy Coupling with Bottleneck (MEC-B) into two distinct optimization problems: Entropy-Bounded Information Maximization (EBIM) for the encoder, and Minimum Entropy Coupling (MEC) for the decoder. Through extensive analysis, we provide a greedy algorithm for EBIM with guaranteed performance, and characterize the optimal solution near functional mappings, yielding significant theoretical insights into the structural complexity of this problem. Furthermore, we illustrate the practical application of MEC-B through experiments in Markov Coding Games (MCGs) under rate limits. These games simulate a communication scenario within a Markov Decision Process, where an agent must transmit a compressed message from a sender to a receiver through its actions. Our experiments highlight the trade-offs between MDP rewards and receiver accuracy across various compression rates, showcasing the efficacy of our method compared to conventional compression baseline.

Figures (16)

• Figure 1: An example for Theorem \ref{['thm:neighbor']}.
• Figure 2: Solutions to the EBIM problem for $p_X=[0.7, 0.2, 0.1]$. Left: brute force solution. Right: application of the transformations from Theorem \ref{['thm:neighbor']} to each deterministic mapping (dashed lines) and selection of solutions with maximal mutual information for each $R$ value (thick solid line). This strategy effectively recovers optimal solutions, aligning with those found by brute force in this case.
• Figure 3: The structure of a Markov Coding Game with Rate Limit.
• Figure 4: The trade-off between average MDP reward vs. receiver's accuracy, navigated by varying the value of $\beta$. Left: using our search algorithm for compression (Algorithm \ref{['alg:search']}), Right: using uniform quantization in Algorithm \ref{['alg:uniformquant']}. The message size is 512 with a uniform prior, and each data point is averaged over 200 episodes.
• Figure 5: Evolution of message belief over time, for various values of $\beta$ and rate budget, using our search algorithm for compression in Algorithm \ref{['alg:search']} vs. uniform quantization in Algorithm \ref{['alg:uniformquant']}.

Theorems & Definitions (21)

• Lemma 1
• Theorem 1
• Remark 1
• Theorem 2
• Theorem 3
• Lemma 1
• proof
• Theorem 3
• proof
• Remark 2