Adversarial Dependence Minimization

TL;DR

The paper tackles nonlinear dependencies among learned embedding dimensions by introducing Adversarial Dependence Minimization (ADMin), a differentiable, scalable training objective where small dependency predictors try to reconstruct a dimension from the others while an encoder counteracts to minimize dependencies. Through a standardized formulation and a margin variant, ADMin converges to minimally dependent representations and supports practical extensions such as nonlinear PCA (PICA), improved classifier generalization, and prevention of dimensional collapse in self-supervised learning. Empirical results show convergence on TinyImageNet and ImageNet, reduced distance correlations, and improved generalization on synthetic and real-world tasks, though SSL on ImageNet indicates room for improvement relative to state-of-the-art methods. The approach provides a flexible regularizer that can be integrated into various learning paradigms to encourage richer, less redundant representations with potential benefits for generalization and robustness.

Abstract

Many machine learning techniques rely on minimizing the covariance between output feature dimensions to extract minimally redundant representations from data. However, these methods do not eliminate all dependencies/redundancies, as linearly uncorrelated variables can still exhibit nonlinear relationships. This work provides a differentiable and scalable algorithm for dependence minimization that goes beyond linear pairwise decorrelation. Our method employs an adversarial game where small networks identify dependencies among feature dimensions, while the encoder exploits this information to reduce dependencies. We provide empirical evidence of the algorithm's convergence and demonstrate its utility in three applications: extending PCA to nonlinear decorrelation, improving the generalization of image classification methods, and preventing dimensional collapse in self-supervised representation learning.

Figures (8)

• Figure 1: Illustration of the joint and marginal distributions for different types of dependencies between random variables ${\textnormal{x}}_1 \in \mathcal{U}(-1,1)$ and $X_2$. The variables are linearly uncorrelated ($\rho = 0$) in both (b) and (c) but are independent only in (c).
• Figure 2: Illustration of the adversarial dependence minimization architecture. The dependency predictors minimize the reconstruction error by learning how dimensions relate, while the encoder maximizes the error by reducing dependencies.
• Figure 3: Convergence analysis on TinyImageNet for linear and non-linear dependency predictors. Left: the reconstruction MSE converges to one. Right: logarithmic plot of the average absolute value of the Pearson correlation on the validation set: correlation decreases over time.
• Figure 4: Learned representations ${\mathbf{z}}$. The colors indicate the value of the original latent factors ${\textnormal{v}}_1$ (left) and ${\textnormal{v}}_2$ (right).
• Figure 5: Influence of the loss margin on the correlation and accuracy for classification models trained on the Clevr-4 dataset. The models are trained on the shape taxonomy and the kNN accuracy is averaged over the three remaining taxonomies.

Theorems & Definitions (5)

• Example 1
• Proposition 4.1
• Example 2
• Example 3
• Example 4