Stable and Convexified Information Bottleneck Optimization via Symbolic Continuation and Entropy-Regularized Trajectories

TL;DR

The paper tackles instability in Information Bottleneck optimization caused by phase transitions as the trade-off parameter $\beta$ varies. It introduces a convexified IB objective with a strictly convex $u(\cdot)$ on $I(X;Z)$ and an entropy regularizer $-\epsilon H(Z|X)$, paired with a predictor–corrector continuation based on an implicit ODE to track the encoder $p(z|x)$ across increments in $\beta$. The main contributions are proving convexity and uniqueness of the IB solution path, developing Hessian-aware continuation, and demonstrating smoother information trajectories that match standard IB performance at the optimum but without discontinuities. The approach yields stable, continuous representations suitable for dynamic or sensitive settings and offers a pathway to extending IB ideas to Gaussian and variational formulations in neural networks.

Abstract

The Information Bottleneck (IB) method frequently suffers from unstable optimization, characterized by abrupt representation shifts near critical points of the IB trade-off parameter, beta. In this paper, I introduce a novel approach to achieve stable and convex IB optimization through symbolic continuation and entropy-regularized trajectories. I analytically prove convexity and uniqueness of the IB solution path when an entropy regularization term is included, and demonstrate how this stabilizes representation learning across a wide range of \b{eta} values. Additionally, I provide extensive sensitivity analyses around critical points (beta) with statistically robust uncertainty quantification (95% confidence intervals). The open-source implementation, experimental results, and reproducibility framework included in this work offer a clear path for practical deployment and future extension of my proposed method.

Figures (11)

• Figure 1: Smooth evolution of the IB solution for the 2x2 binary symmetric channel. Standard IB (orange solid = $I(Z;Y)$, orange dashed = $I(X;Z)$) exhibits an abrupt phase transition: no information is transmitted until a critical $\beta \approx 1.6$, where it suddenly jumps to a higher $I(Z;Y)$. In contrast, Entropy-Regularized IB (magenta/red curves, with $\epsilon=0.1$) shows a gradual increase in mutual informations, avoiding any discontinuous jump. Convexified IB (with $u(t)=t^2$, not shown separately) similarly yields a smooth trajectory, essentially overlapping with the entropy-regularized curve in this simple case. All approaches converge to the same maximal $I(Z;Y) \approx 0.53$ bits as $\beta$ becomes large (full information). The entropy term causes $I(X;Z)$ to remain slightly below $1$ bit even at high $\beta$ (since a tiny amount of randomness is retained). Overall, the proposed methods achieve a stable information trade-off curve.
• Figure 2: Phase transition detection in the BSC example. The orange curve shows the smallest Hessian eigenvalue of the standard IB objective as $\beta$ increases. It drops to zero at $\beta \approx 1.57$, indicating a pitchfork bifurcation where the trivial encoder loses stability. My method monitors this eigenvalue and applies convexification/regularization to avoid crossing into negative curvature. As a result, the modified IB solver never actually reaches $\lambda_{\min}=0$ – the trajectory veers away from the would-be bifurcation. The dashed line at eigenvalue $0$ is the theoretical bifurcation threshold. Detecting this crossing allows the algorithm to adjust (e.g., increase entropy regularization) preemptively. In practice, standard IB would exhibit an abrupt jump in representation at this point, whereas the convexified/entropy-regularized IB transitions through smoothly.
• Figure 3: A "grand tour" of IB from $\beta=0 \to 10$. Top: Info-plane plot, nearly linear for the convexified path. Bottom: $\lambda_{\min}$ vs. $\beta$, rising from $\approx 0.7$ to $\gg 100$. No hidden instabilities after $\beta \approx 3$.
• Figure 4: Heatmaps $p(z \mid x)$ at some fixed $\beta$. Std IB has crisp vertical stripes, Convex IB slightly more gradual, Entropy-Reg close to Standard but less "binary."
• Figure 5: Snapshots of $p(z \mid x)$ over $\beta$ steps. Row 1 fuzzy $\to$ 2 clusters $\to$ 3 clusters, row 2 saturates to near-deterministic. Shows no random flips, exactly at $\lambda_{\min}=0$ we see a new cluster appear.