Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Disentangled Representation Learning with Transmitted Information Bottleneck

Zhuohang Dang, Minnan Luo, Chengyou Jia, Guang Dai, Jihong Wang, Xiaojun Chang, Jingdong Wang

TL;DR

DisTIB introduces a novel transmitted information bottleneck framework to disentangle label-related information from sample-specific information in representations. By using two Bayesian networks to model information compression and preservation, and deriving variational bounds for all mutual information terms, it achieves stable optimization and theoretical guarantees of optimal disentanglement. Empirically, DisTIB improves adversarial robustness, generalization, and few-shot and fine-grained learning across multiple datasets, while enabling disentangled generation and qualitative attribute control. The work demonstrates that explicit disentanglement constraints can be avoided in favor of a principled variational objective that yields better information control and stable training, with broad applicability to real-world tasks.

Abstract

Encoding only the task-related information from the raw data, \ie, disentangled representation learning, can greatly contribute to the robustness and generalizability of models. Although significant advances have been made by regularizing the information in representations with information theory, two major challenges remain: 1) the representation compression inevitably leads to performance drop; 2) the disentanglement constraints on representations are in complicated optimization. To these issues, we introduce Bayesian networks with transmitted information to formulate the interaction among input and representations during disentanglement. Building upon this framework, we propose \textbf{DisTIB} (\textbf{T}ransmitted \textbf{I}nformation \textbf{B}ottleneck for \textbf{Dis}entangled representation learning), a novel objective that navigates the balance between information compression and preservation. We employ variational inference to derive a tractable estimation for DisTIB. This estimation can be simply optimized via standard gradient descent with a reparameterization trick. Moreover, we theoretically prove that DisTIB can achieve optimal disentanglement, underscoring its superior efficacy. To solidify our claims, we conduct extensive experiments on various downstream tasks to demonstrate the appealing efficacy of DisTIB and validate our theoretical analyses.

Disentangled Representation Learning with Transmitted Information Bottleneck

TL;DR

DisTIB introduces a novel transmitted information bottleneck framework to disentangle label-related information from sample-specific information in representations. By using two Bayesian networks to model information compression and preservation, and deriving variational bounds for all mutual information terms, it achieves stable optimization and theoretical guarantees of optimal disentanglement. Empirically, DisTIB improves adversarial robustness, generalization, and few-shot and fine-grained learning across multiple datasets, while enabling disentangled generation and qualitative attribute control. The work demonstrates that explicit disentanglement constraints can be avoided in favor of a principled variational objective that yields better information control and stable training, with broad applicability to real-world tasks.

Abstract

Encoding only the task-related information from the raw data, \ie, disentangled representation learning, can greatly contribute to the robustness and generalizability of models. Although significant advances have been made by regularizing the information in representations with information theory, two major challenges remain: 1) the representation compression inevitably leads to performance drop; 2) the disentanglement constraints on representations are in complicated optimization. To these issues, we introduce Bayesian networks with transmitted information to formulate the interaction among input and representations during disentanglement. Building upon this framework, we propose \textbf{DisTIB} (\textbf{T}ransmitted \textbf{I}nformation \textbf{B}ottleneck for \textbf{Dis}entangled representation learning), a novel objective that navigates the balance between information compression and preservation. We employ variational inference to derive a tractable estimation for DisTIB. This estimation can be simply optimized via standard gradient descent with a reparameterization trick. Moreover, we theoretically prove that DisTIB can achieve optimal disentanglement, underscoring its superior efficacy. To solidify our claims, we conduct extensive experiments on various downstream tasks to demonstrate the appealing efficacy of DisTIB and validate our theoretical analyses.
Paper Structure (44 sections, 2 theorems, 8 equations, 9 figures, 4 tables, 1 algorithm)

This paper contains 44 sections, 2 theorems, 8 equations, 9 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Disentangled Representation Learning
  4. Information Bottleneck
  5. Methodology
  6. Transmitted Information Bottleneck
  7. Problem Definition
  8. Variable Bayesian Networks
  9. Formulation
  10. Theoretical Analyses
  11. DisTIB Optimization and Inference
  12. Compression Term
  13. Sufficiency Term
  14. Prediction Term
  15. Model Inference
  16. ...and 29 more sections

Key Result

Theorem 1

Given Bayesian networks $\mathcal{G}$ over variables $\bm{\mathcal{X}}=\{X_1,\cdots,X_n\}$, let $\mathbf{P a}_{X_i}^\mathcal{G}$ denotes parents of $X_i$ in $\mathcal{G}$, the joint distribution can be factorized as $p(\bm{\mathcal{X}})=\Pi_{i=1}^n p(X_i | \mathbf{P a}_{X_i}^\mathcal{G})$, thus the where $I(X_i;\mathbf{P a}_{X_i}^\mathcal{G})$ is the transmitted information, an extension of mutua

Figures (9)

  • Figure 1: Illustration of Bayesian networks for information compression and preservation. The white and gray nodes denote input and target disentangled variables, respectively. The arrow indicates the direction of information flow, i.e., variable interactions, during the disentangled representation learning.
  • Figure 2: Illustration of DisTIB's training procedure on MNIST dataset, where disentangled variables gradually converge to their optimal value. In detail, the solid lines represent the estimation of mutual information between representations. The blue dashed line represents entropy of labels, $H(Y)$, which is the optimal value for the disentangled representation mutual information $I(X;A)$ and $I(A;Y)$.
  • Figure 3: Visualization of disentangled sample generation, where samples are generated by the label-related information $A$ of the top row and sample-exclusive information $Z$ of the leftmost column. The top row and leftmost column images come from the dataset and the diagonal images show reconstructions.
  • Figure 4: Illustration of applying our DisTIB to disentangle facial attributes. Each row displays the generation results of a specific facial attribute interpolation, where the leftmost/rightmost image has an attribute value of 0/1; intermediate images represent interpolated attributes with a step size of 0.1.
  • Figure 5: The performance-compression trade-off on MNIST, where DisTIB is represented as dots due to its convergence to optimal disentanglement.
  • ...and 4 more figures

Theorems & Definitions (4)

  • Theorem 1
  • proof
  • Theorem 2
  • proof