The geometry of efficient codes: how rate-distortion trade-offs distort the latent representations of generative models

TL;DR

This work investigates how rate-distortion trade-offs shape the geometry of latent representations in generative models, using a $eta$-VAE on a carefully designed Corridors dataset with two orthogonal generative factors. By varying model capacity, data distributions, and task objectives, the study identifies three high-signal distortions—prototypization, specialization, and orthogonalization—that can co-exist and reconfigure latent spaces. Experiment 1 shows that capacity limits and data biases distort disentanglement, sometimes increasing latent dimensionality as a means to separate rare from frequent stimuli. Experiment 2 demonstrates that adding classification tasks reshapes representations, with high capacity enabling task-specific latent channels and low capacity driving strong orthogonalization and clustering; joint training across multiple tasks yields a rich, multi-faceted latent geometry. Collectively, the results illuminate how normative rate-distortion constraints sculpt internal representations in both artificial systems and potentially living neural systems, offering a framework to interpret and reverse-engineer brain-like compression and to design more robust generative models under resource constraints.

Abstract

Living organisms rely on internal models of the world to act adaptively. These models, because of resource limitations, cannot encode every detail and hence need to compress information. From a cognitive standpoint, information compression can manifest as a distortion of latent representations, resulting in the emergence of representations that may not accurately reflect the external world or its geometry. Rate-distortion theory formalizes the optimal way to compress information while minimizing such distortions, by considering factors such as capacity limitations, the frequency and the utility of stimuli. However, while this theory explains why the above factors distort latent representations, it does not specify which specific distortions they produce. To address this question, here we investigate how rate-distortion trade-offs shape the latent representations of images in generative models, specifically Beta Variational Autoencoders ($β$-VAEs), under varying constraints of model capacity, data distributions, and task objectives. By systematically exploring these factors, we identify three primary distortions in latent representations: prototypization, specialization, and orthogonalization. These distortions emerge as signatures of information compression, reflecting the model's adaptation to capacity limitations, data imbalances, and task demands. Additionally, our findings demonstrate that these distortions can coexist, giving rise to a rich landscape of latent spaces, whose geometry could differ significantly across generative models subject to different constraints. Our findings contribute to explain how the normative constraints of rate-distortion theory shape the geometry of latent representations of generative models of artificial systems and living organisms.

Figures (19)

• Figure 1: Formal framework adopted in this study, based on rate-distortion theory (RDT) shannon1959coding. (A) Trade-off between rate (or available resources) and distortion. (B) The goal of RDT is to find the minimum distortion function given a constraint on the available resources. The loss of a $\beta$ variational autoencoder has the same shape as the Lagrangian of this minimization problem. (C) Architecture of the network used in experiments: it is a classical $\beta$-VAE that can be optionally augmented with $n$ classifiers. In most of our experiments $n$ is equal $0$ or $1$. The classifiers can be both linear or non-linear. The loss of the classifiers is added to the loss of the $\beta$-VAE as described in panel B. (D) Stimuli with higher utility are encoded more faithfully. Utility of a stimulus is related to its likelihood or to its relevance with respect to a task. This panel shows that, under strong resource constraints (small rate), stimuli with small probability of occurrence are ignored. (E) This panel shows how faithfully a stimulus is reconstructed under different rate and relevance conditions: at small rates, two stimuli with small relevance are collapsed into the same representation while details about stimuli with high relevance are still preserved even at small rates.
• Figure 2: The Corridors dataset. The figure shows 21 example images in the "corridors" dataset used for this study. Each figure comprises two white corridors, placed at the upper and at the lower parts of the image, with a white horizontal line that is common to all the images. Each corridor is a noisy vertical line of white pixels, whose true center is in one of 13 $x$ positions, coded from 0 (left) to 12 (right). For example, in the first image, the upper corridor is in the position $x_{UC}=0$ and the lower corridor is in the position $x_{LC}=12.$. The corridor positions are reported on top of each image.
• Figure 3: Reconstruction loss, all the models. For ease of reading, a brief description of each model along with its label is reported in the legend. (A) Experiment 1. The figure illustrates that increasing capacity reduces reconstruction loss. The trend is similar for the baseline $\beta$-VAE model that is trained with a balanced dataset and for the two models (E1M1 - E1M2) that are trained with unbalanced datasets. (B) Experiment 2. The figure illustrates that at any given capacity, the baseline model has a smaller reconstruction loss compared to the hybrid models that are additionally trained to solve classification tasks (E2M1 - E2M5). Furthermore, at any given capacity, reconstruction loss changes across the different tasks and is worst for the E2M5 model, which addresses four classification tasks simultaneously. See the main text for explanation.
• Figure 4: Comparison of the latent representations of the baseline model and the model E1M1. The E1M1 model is trained on an unbalanced dataset, in which images with $x_{LC} \leq 6$ (orange) are 10 times more frequent than images with $x_{LC} > 6$ (green). (A, F) 2D projections of the 5D embeddings learned by the baseline model at high ($C_{\max}=10$ nats) and low ($C_{\max}=0.3$ nats) capacity, respectively. (C, H) 2D projections of the 5D embeddings learned by the hybrid E1M1 model trained at high ($C_{\max}=10$ nats) and low ($C_{\max}=0.3$ nats) capacity, respectively. (B, G) Activation patterns of the 5 latent channels of the baseline model at high and low capacity, respectively. Each of the five heat-maps is computed as in sec. \ref{['sec:neuron_activity']}. (D, I) Activation patterns of the 5 latent channels of the E1M1 model at high and low capacity, respectively. Each heat-map is computed as in Section \ref{['sec:neuron_activity']}. (E, J) Measure of distortions in the latent representations of the hybrid E1M1 model compared to the baseline model at high and low capacity, respectively. (K) Measure of distortions in the latent representation of the baseline model induced by the reduction of the encoding capacity from high to low. (L) Measure of distortions in the latent representation of the E1M1 model induced by the reduction of the encoding capacity from high to low. Distortion matrices are computed as described in Section \ref{['sec:compare_lr']}.
• Figure 5: Comparison of the latent representations of the baseline model and the hybrid model E1M2. The E1M2 model is trained on an unbalanced dataset in which images with $x_L = x_U$ (orange) are 10 times more frequent than images with $x_L \neq x_U$ (green). (A, F) 2D projections of the 5D embeddings learned by the baseline model at high ($C_{\max}=10$ nats) and low ($C_{\max}=0.3$ nats) capacity, respectively. (C, H) 2D projections of the 5D embeddings learned by the hybrid E1M2 model trained at high ($C_{\max}=10$ nats) and low ($C_{\max}=0.3$ nats) capacity, respectively. (B, G) Activation patterns of the 5 latent channels of the baseline model at high and low capacity, respectively. Each of the five heat-maps is computed as in sec. \ref{['sec:neuron_activity']}. (D, I) Activation patterns of the 5 latent channels of the E1M2 model at high and low capacity, respectively. Each heat-map is computed as in Section \ref{['sec:neuron_activity']}. (E, J) Measure of distortions in the latent representations of the hybrid E1M2 model compared to the baseline model at high and low capacity, respectively. (K) Measure of distortions in the latent representation of the baseline model induced by the reduction of the encoding capacity from high to low. (L) Measure of distortions in the latent representation of the E1M2 model induced by the reduction of the encoding capacity from high to low. Distortion matrices are computed as described in Section \ref{['sec:compare_lr']}.