Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis

TL;DR

The paper questions representational optimism by showing that two networks can produce identical outputs but rely on radically different internal representations, with SGD-trained models exhibiting fractured entangled representations (FER) while open-ended Picbreeder-evolved networks tend toward unified factored representations (UFR). It introduces FER as a potential barrier to generalization, creativity, and continual learning, and UFR as an aspirational modular structure; it uses CPPNs and visualizations to compare internal representations, including weight-sweep analyses and PCA, across Picbreeder and SGD. The authors discuss evidence of FER-like behaviors in large models (e.g., LLMs and image generators) and explore factors that influence representation formation, such as learning order, architectural choices, data, and open-ended search, while suggesting that open-ended, curricula-informed approaches might yield more robust representations. The work serves as a call to broaden the focus beyond output accuracy to the quality of internal representations, with implications for future training paradigms, mechanistic interpretability, and the design of scalable, adaptable AI systems.

Abstract

Much of the excitement in modern AI is driven by the observation that scaling up existing systems leads to better performance. But does better performance necessarily imply better internal representations? While the representational optimist assumes it must, this position paper challenges that view. We compare neural networks evolved through an open-ended search process to networks trained via conventional stochastic gradient descent (SGD) on the simple task of generating a single image. This minimal setup offers a unique advantage: each hidden neuron's full functional behavior can be easily visualized as an image, thus revealing how the network's output behavior is internally constructed neuron by neuron. The result is striking: while both networks produce the same output behavior, their internal representations differ dramatically. The SGD-trained networks exhibit a form of disorganization that we term fractured entangled representation (FER). Interestingly, the evolved networks largely lack FER, even approaching a unified factored representation (UFR). In large models, FER may be degrading core model capacities like generalization, creativity, and (continual) learning. Therefore, understanding and mitigating FER could be critical to the future of representation learning.

Figures (25)

• Figure 1: Overview. An open-ended search process (left) from the Picbreeder experiment yielded a neural network that outputs the image of a skull. Conventional SGD (right) is able to learn to produce precisely the same image, but by following a completely different path through function space during learning. However, peering inside these networks with identical output behavior reveals a radically different style of representation, raising questions about the adaptability of networks (including large models) trained through conventional SGD, which can impact generalization, creativity, and (continual) learning.
• Figure 2: Example compositional pattern producing network (CPPN). A CPPN is a neural network that takes as input the $x$, $y$, and $d=\sqrt{x^2+y^2}$ coordinates of a pixel and outputs the pixel's $h$, $s$, and $v$ color values. By sweeping over all possible $x$ and $y$ pixel locations, an image can be generated, visualizing the CPPN's output behavior for all inputs. Importantly, the activation of each hidden neuron over all inputs can be visualized as an image (shown in the red–blue insets, where red and blue indicates low and high activation), thus enabling inspection of how the CPPN internally represents and builds up its output behavior. In this paper, CPPNs serve as an analogy for large AI models like LLMs.
• Figure 3: Picbreeder. Picbreeder secretan2008picbreedersecretan2011picbreeder was a system that allowed humans to breed CPPNs based on their desires, using the NEAT evolutionary algorithm stanley2002evolving. (a) This Picbreeder subphylogeny shows the open-ended nature of Picbreeder. It allowed users to keep generating interesting images from each other indefinitely. (b) Notable examples of CPPNs discovered by Picbreeder are in effect needles in a haystack in the space of all possible CPPNs.
• Figure 4: The Picbreeder skull CPPN and its conventional SGD equivalent. The image on the left (a) is the original discovery of the Picbreeder skull, represented by an evolved CPPN. On the right (b), the skull produced by SGD when trained to output the Picbreeder skull is effectively identical, showing that SGD can easily learn the same behavior output as training targets. However, despite their outward similarity, their internal representations are radically different (as discussed next in the main text). Appendix \ref{['sec:relucppn']} shows that SGD results also hold for ReLU-only architectures.
• Figure 5: Internal representations of the Picbreeder and conventional SGD skull CPPNs. This figure illustrates how each CPPN processes its input through its neural circuitry to generate its output. Each image visualizes the latent representation of all inputs at a specific layer and neuron (red and blue represent low and high activation, respectively). Notice the stark contrast between the Picbreeder CPPN (a), which represents the skull through organized circuitry, and the conventional SGD CPPN (b), which represents the same skull through disorganized patchwork. A green border indicates the latent representation is novel (not seen in previous layers). The layerized version of the Picbreeder CPPN includes only 24 novel representations, highlighting the efficiency of its encoding.