An Explicit Description of Extreme Points of the Set of Couplings with Given Marginals: with Application to Minimum-Entropy Coupling Problems

TL;DR

The paper provides a complete structural description of extreme points in the set of couplings with fixed marginals ${\cal C}(\mathbf p,\mathbf q)$, proving that ${\cal C}_e(\mathbf p,\mathbf q)={\mathscr C}(\mathbf p,\mathbf q)$ where extreme points correspond to forest-structured supports. It then shows that the minimum-entropy coupling, and more generally the minimum of any strict Schur-concave function, is achieved within this finite extreme-point set, enabling exact solutions for Shannon, Rényi, and Tsallis-type entropies, and generalizing to multi-marginal problems. An explicit Min Entropy Coupling Algorithm enumerates all candidate forest-structures via a structure matrix ${\cal A}(V)$, solves linear systems to recover feasible couplings, and selects the minimum entropy. The results illuminate the geometry of the coupling polytope and provide a practical, rigorous approach for exact probabilistic inference and information-theoretic optimization with fixed marginals.

Abstract

Given probability distributions ${\bf p}=(p_1,p_2,\ldots,p_m)$ and ${\bf q}=(q_1,q_2,\ldots, q_n)$ with $m,n\geq 2$, denote by ${\cal C}(\bf p,q)$ the set of all couplings of $\bf p,q$, a convex subset of $\R^{mn}$. Denote by ${\cal C}_e({\bf p},{\bf q})$ the finite set of all extreme points of ${\cal C}(\bf p,q)$. It is well known that, as a strictly concave function, the Shannan entropy $H$ on ${\cal C}(\bf p,q)$ takes its minimal value in ${\cal C}_e({\bf p},{\bf q})$. In this paper, first, the detailed structure of ${\cal C}_e({\bf p},{\bf q})$ is well specified and all extreme points are enumerated by a special algorithm. As an application, the exact solution of the minimum-entropy coupling problem is obtained. Second, it is proved that for any strict Schur-concave function $Ψ$ on ${\cal C}(\bf p,q)$, $Ψ$ also takes its minimal value on ${\cal C}_e({\bf p},{\bf q})$. As an application, the exact solution of the minimum-entropy coupling problem is obtained for $(Φ,\hbar)$-entropy, a large class of entropy including Shannon entropy, Rényi entropy and Tsallis entropy etc. Finally, all the above are generalized to multi-marginal case.

Figures (2)

• Figure 1: (a) is the graph $G_{m,n}$ with $m=n=10$, it has $n^2$ vertices and $2n\binom{n}{2}=n^3-n$ edges. In $G_{10,10}$, $\gamma_1$ is a circuit, but $\gamma=\{(1,1),(1,4),(1,7),(1,1)\}$ is not a circuit. (b) is the graph $\bar{G}_{m,n}$ introduced by CK with $m=n=6$, which is a bipartite graph with $2n$ vertices and $n^2$ edges, where $\gamma_2$ is a circuit in $\bar{G}_{m,n}$.
• Figure 2: An example to show the structure of $Q$: a) $\bullet$, $\star$ is the new position of a point in $V_2(P)$, $V_3(P)$ respectively, $\circ$'s are zeros. $|V(Q)|=|V(P)|$ and $V(Q)=\cup_{k=0}^{\tau}\gamma_k$, $\tau=4$. For each $1\leq k\leq 4$, $u_k$ is the beginning vertex of $\gamma_k$, $v_k$ ($\notin V(Q)$) is the beginning vertex of $\bar{\gamma}_k$, and $\gamma_0\cup\bar{\gamma}_1\cup\cdots\cup\bar{\gamma}_4$ forms a continuous directed path from $(1,1)$ to $(m,n)=(27,26)$. b) In the definition of $\gamma_0$, one has $\xi_0=2\zeta_0-1=7$, $k_0=2z_0=4$, $s_1=s_2=s_4=s_6=1$, $s_3=s_5=s_7=2$, $i(1)=3,\ i(2)=6,\ i(3)=9$, $j(1)=3,\ j(2)=5,\ j(3)=7$. Before we define $\gamma_1$, we define $(i(4),j(4))=(11,8)=v_1$. c) The minimum-entropy coupling possesses nice local features, for example, by Theorem \ref{['t2']}, the subpath $\gamma_0'$$=\{(1,1),(3,1),(3,3),(6,3),(6,5),(9,5),(9,7),(11,7)\}$, which forms the skeleton of $\gamma_0$, behaves supper-Fibonacci, i.e. $q_{1,1}+q_{3,1}\leq q_{3,3}$, $q_{3,1}+q_{3,3}\leq q_{6,3}$,…, $q_{9,5}+q_{9,7}\leq q_{11,7}$.

Theorems & Definitions (24)

• Definition 1.1
• Remark 1.2
• Definition 1.3
• Remark 1.4
• Proposition 1.5
• Proposition 1.6
• Remark 1.7
• Lemma 1.8
• Definition 1.9
• Proposition 1.10