Lossy Common Information in a Learnable Gray-Wyner Network

TL;DR

The paper tackles redundancy across vision tasks by learning a Gray-Wyner-like three-channel codec that separates a common representation $Y_0$ from private task representations, under distortions $(D_1,D_2)$. It develops bounds for lossy common information via interaction information, and introduces a transmit-receive tradeoff optimization with a tunable parameter $\beta$ to navigate $R_t$ and $R_r$, supported by a learnable architecture that uses entropy-model rate terms and an augmented loss to encourage common-channel usage. The proposed network demonstrates substantial reductions in transmit rate while maintaining competitive task performance across synthetic data and benchmarks like Cityscapes and COCO, with BD-rate improvements on the order of tens of percent and up to around $-81.6\%$ in transmit-rate compared to single-task baselines. These results validate revisiting Gray-Wyner theory for task-driven representation learning and suggest extensions to more tasks and scalable architectures for distributed inference and storage.

Abstract

Many computer vision tasks share substantial overlapping information, yet conventional codecs tend to ignore this, leading to redundant and inefficient representations. The Gray-Wyner network, a classical concept from information theory, offers a principled framework for separating common and task-specific information. Inspired by this idea, we develop a learnable three-channel codec that disentangles shared information from task-specific details across multiple vision tasks. We characterize the limits of this approach through the notion of lossy common information, and propose an optimization objective that balances inherent tradeoffs in learning such representations. Through comparisons of three codec architectures on two-task scenarios spanning six vision benchmarks, we demonstrate that our approach substantially reduces redundancy and consistently outperforms independent coding. These results highlight the practical value of revisiting Gray-Wyner theory in modern machine learning contexts, bridging classic information theory with task-driven representation learning.

Figures (10)

• Figure 1: Diagram of the Gray-Wyner Network and a lower bound on its achievable region. The lower bound is given by planes $A$, $B$, and $C$. Plane $A$ corresponds to the Pangloss, the contour of the achievable region that achieves ${R_{X_1,X_2}(D_1,D_2)}$. Points with $R_0 = C(X_1,X_2;D_1,D_2)$ are found on it. Planes $B$ and $C$ achieve ${R_0 + R_1 = R_{X_1}(D_1)}$ and ${R_0 + R_2 = R_{X_2}(D_2)}$, respectively. On the intersection of both of these planes, we can find a point with $R_0 = K(X_1,X_2;D_1,D_2)$. A trace connecting both points offers tradeoffs between the transmit and receive rates. If the mutual information between tasks, at distortions $(D_1, D_2)$, is separable, both points coincide at the now achievable Separable point. We show that points close to it, whether achievable or not, serve as bounds for both lossy common information measures.
• Figure 2: Architecture overview. The parameters for the probability distribution of the target representations are represented as ${\tilde{P}_{Y_{\{0,1,2\}}}}$. Grey binary blocks denote bitstreams with rates ${R_{\{0,1,2\}}}$. For the particular choice of analysis and synthesis transforms used in the experimental evaluation, the grey bars on the corresponding blocks indicate the number of downsample or upsample operations.
• Figure 3: Rate-distortion curves for different methods, optimized for transmit or receive rate, or a mixture of both. BD-rates bjontegaard2001calculation are computed with respect to the method with no assigned score. Bits-per-pixel (BPP) is the bitrate scaled by $1/64 \times 64$.
• Figure 4: Validation rate-accuracy curves for colored MNIST on 3 different PMFs. The top-1 classification accuracies are added and shown on the vertical axis. The total bitrate is shown as rate. BD-rates are computed with respect to the Dependent PMF.
• Figure 5: Rate-accuracy curves of the proposed method against the Joint and Independent baselines. The tasks in (a) are reported for the validation set of Cityscapes. The tasks in (b) are reported for the validation set of COCO 2017. The transmit and receive rates of the proposed method are included. BD-rates are computed with respect to the Joint method. The task performances are added. The depth RMSE is scaled so its inverse is in a similar scale as the segmentation mean intersection over union (mIoU). The detection performances are measured by the mean average precision (mAP). The Uncompressed lines correspond to the original performances of the pre-trained task models.

Theorems & Definitions (15)

• Theorem 1
• Theorem 2
• Lemma 1
• proof
• Theorem 2
• proof
• Theorem 2
• proof
• Definition 1
• Lemma 2