Cross-Entropy Is All You Need To Invert the Data Generating Process

TL;DR

This work develops a theoretical framework showing that cross-entropy-based supervised classification learns ground-truth latent factors up to a linear transformation in a $d$-dimensional space. Building on nonlinear ICA and the DIET instance-discrimination pipeline, it introduces a cluster-centric DGP and proves identifiability for both latent variables and cluster vectors, with supervised classification emerging as a special case. Empirical validation on synthetic data, DisLib, and ImageNet-X demonstrates that latent factors are recoverable and linearly decodable from representations learned through standard classification. The results offer a cohesive explanation for the observed linearity in neural representations and transferability, providing a principled bridge between self-supervised ICA insights and supervised deep learning practice.

Abstract

Supervised learning has become a cornerstone of modern machine learning, yet a comprehensive theory explaining its effectiveness remains elusive. Empirical phenomena, such as neural analogy-making and the linear representation hypothesis, suggest that supervised models can learn interpretable factors of variation in a linear fashion. Recent advances in self-supervised learning, particularly nonlinear Independent Component Analysis, have shown that these methods can recover latent structures by inverting the data generating process. We extend these identifiability results to parametric instance discrimination, then show how insights transfer to the ubiquitous setting of supervised learning with cross-entropy minimization. We prove that even in standard classification tasks, models learn representations of ground-truth factors of variation up to a linear transformation. We corroborate our theoretical contribution with a series of empirical studies. First, using simulated data matching our theoretical assumptions, we demonstrate successful disentanglement of latent factors. Second, we show that on DisLib, a widely-used disentanglement benchmark, simple classification tasks recover latent structures up to linear transformations. Finally, we reveal that models trained on ImageNet encode representations that permit linear decoding of proxy factors of variation. Together, our theoretical findings and experiments offer a compelling explanation for recent observations of linear representations, such as superposition in neural networks. This work takes a significant step toward a cohesive theory that accounts for the unreasonable effectiveness of supervised deep learning.

Figures (6)

• Figure 1: DIET ibrahim_occams_2024 learns identifiable features: given $N$ samples and a $d-$dimensional latent representation, DIET learns a linear ${(N\times d)-}$dimensional classification head on top of a nonlinear encoder $\f$ through an instance discrimination objective \ref{['eq:DIET_loss_x_main']}. For unit-normalized $\f(\x_{n}),$ DIET maps samples and their augmentations close to the cluster vector $\vv_{\c}$ corresponding to the class---as if sampled from a distribution, centered around $\vv_{\c}$. For duplicate samples, i.e., matching class labels, the corresponding rows of $\W$ will be the same, as shown for $\x_1$ and $\x_i$ with $\w[1]=\w[i]$.
• Figure 2: The simplified genealogy of cross-entropy-based classification methods (cf. \ref{['tab:method_comp']} for details): The labeled arrows express how to go from general to special methods. (a) The most general auxiliary-variable framework, gencl hyvarinen_nonlinear_2019, yields tcl hyvarinen_unsupervised_2016 as the special case when the latent conditional is assumed to come from an exponential family (of order one) with a scalar auxiliary variable; (b) relates to non-unit-normalized DIET by further restricting the latent conditional to a distribution; (c) if the neural network used in InfoNCE is partitioned into a linear classifier head and a backbone, the marginal is assumed to be a instead of uniform, we get the unit-normalized version of DIET; (d) if the labeling function in DIET is assumed to assign the semantic class labels to the samples, we get classic supervised training
• Figure 3: Approximate identifiability on ImageNet-X against a random (shuffled) baseline: Using ImageNet-X idrissi2022imagenetx, we test how well linear decoders are able to predict each latent from the second-to-last layer of different models, i.e., when the classification head is discarded. We train a linear classifier on the features, and plot the accuracy of predicting different latent variables. As baselines, we also try decoding from the raw input and from the randomly initialized model representations. Error-bars indicate standard error of the mean (SEM) across $10$ seeds of balanced resampling. Asterisks indicate significant $p$-values (against a null hypothesis of $0.5$ chance level accuracy) at an $\kappa = 0.05 / 85$ multiple comparison (Bonferroni) adjusted significance level.
• Figure 4: Quantifying the assumption violation of a Laplace conditional:\ref{['tab:exp_results_ortho', 'tab:sim_results_sup']} show that using a Laplace conditional leads to substantially lower scores. Numbers are averages across 5 seeds. (Left and Middle:) using a generalized normal distribution to "interpolate" between a normal $(\beta=2)$ and a Laplace $(\beta=1)$ distribution for different scale values (denoted as $\alpha$, which is conceptually akin to our concentration $\kappa$) and show the score for recovering $\z$(Left) and $\vv_\c$(Middle). (Right:) The average norm of the representation for the one-dimensional case for different $\beta$ values. As $\beta$ approaches $1$, the average norm increases, indicating a larger spread
• Figure 5: The role of batch size, number of clusters, and fat tails for identifiability:(Left:) Increasing batch size improves scores, counteracting the detrimental effect of more concentrated (higher $\kappa$) conditionals; (Middle:) More clusters improve scores, counteracting the detrimental effect higher dimensional representations (10 clusters for 10 and 20 dimensions violate \ref{['assum:diversity']}; thus, the low score); (Right:) The Laplace distribution leads to low scores due to its fat tails. Experiments with a truncated Laplace conditional (where the support is restricted to $[-\text{Truncation};\text{Truncation}]$) shows that the closer the truncated Laplace distribution is to the Laplace distribution (i.e., with increasing $\text{Truncation}$), scores decrease for all tested scales $\alpha$. Averages and error bars are reported across 5 seeds

Theorems & Definitions (13)

• Theorem 1: Identifiability of latent variables drawn from vMF around cluster vectors. Simplified.
• Theorem 2: Identifiability of latent variables drawn from a vMF around class vectors
• Definition 1: Affine Generator System
• Lemma 1: Properties of affine generator systems
• Theorem \ref{thm:ident_theo_main}C: Identifiability of latents drawn from a vMF around cluster vectors
• Remark
• proof
• Theorem \ref{thm:supervised}C: Identifiability of latent variables drawn from a vMF around class vectors
• proof
• Lemma 2: The cluster-based DIET DGP is sufficiently variable