The World Is Bigger! A Computationally-Embedded Perspective on the Big World Hypothesis

TL;DR

This work reframes continual learning through a computation-embedded lens: an agent is embedded in a universal-local environment, making it implicitly capacity-constrained and compelled to continually adapt. The authors formalize interactivity, an algorithmic-information based measure of an agent's ability to adapt future behavior based on past experience, and develop a model-based RL approach to maximize it. They prove that embedded agents face intrinsic capacity limits and that continual adaptation is necessary to maximize interactivity, a phenomenon that scales with capacity and manifests as a big world-like challenge. Empirically, deep linear policies sustain higher interactivity with increased capacity, while deep nonlinear policies struggle, highlighting a plasticity-stability tension in continual adaptation experiments. The framework offers a principled avenue for evaluating and guiding continual learning dynamics and suggests interactivity as a potential intrinsic reward for exploration in RL.

Abstract

Continual learning is often motivated by the idea, known as the big world hypothesis, that "the world is bigger" than the agent. Recent problem formulations capture this idea by explicitly constraining an agent relative to the environment. These constraints lead to solutions in which the agent continually adapts to best use its limited capacity, rather than converging to a fixed solution. However, explicit constraints can be ad hoc, difficult to incorporate, and may limit the effectiveness of scaling up the agent's capacity. In this paper, we characterize a problem setting in which an agent, regardless of its capacity, is constrained by being embedded in the environment. In particular, we introduce a computationally-embedded perspective that represents an embedded agent as an automaton simulated within a universal (formal) computer. Such an automaton is always constrained; we prove that it is equivalent to an agent that interacts with a partially observable Markov decision process over a countably infinite state-space. We propose an objective for this setting, which we call interactivity, that measures an agent's ability to continually adapt its behaviour by learning new predictions. We then develop a model-based reinforcement learning algorithm for interactivity-seeking, and use it to construct a synthetic problem to evaluate continual learning capability. Our results show that deep nonlinear networks struggle to sustain interactivity, whereas deep linear networks sustain higher interactivity as capacity increases.

Figures (5)

• Figure 1: Comparing the agent and environment in different problem formulations. Each problem formulation differs in the constraints that it imposes on the agent relative to the environment. Traditional RL: A given environment typically has a bounded capacity, but a scalable agent can increase its capacity. Such an agent is unconstrained in principle because its capacity can always be scaled beyond the environment. Universal AI: Both the computationally universal environment and the AIXI agent are unbounded. AIXI is thus unconstrained but not computable. This work: The universal-local environment is unbounded, but it can simulate an embedded agent of any bounded capacity within its state-space. An embedded agent is implicitly constrained because the environment necessarily has greater capacity than any agent contained within it.
• Figure 2: Conway's Game of Life is a cellular automaton and an example of a universal-local environment. The state-space is an infinite 2D grid, $\Xi = \mathbb{Z}^2$, where each cell in the grid takes one of two values, $\Sigma = \{\text{black}, \text{white}\}$. Every cell uses the same local transition function in which the boundary-space consists of its $8$ adjacent neighbours. The blue and green borders (left) correspond to $1$ and $2$ horizon boundary-spaces, which determine the center cell at time-steps $t+1$ (middle) and $t+2$ (right), respectively. Longer-term transition dynamics depend on larger boundary-spaces.
• Figure 3: An illustrative depiction of a computationally-embedded agent interacting with its environment. An embedded automaton, $\mathcal{A} = (\Omega|_X, \Omega|_Y, \Omega|_\Theta, u, \pi)$, represents an agent embedded within the universal-local environment. Left: The agent is simulated by a boundaried Markov process within the environment, in which it iteratively receives an input from the environment, $x \in \Omega|_X$, produces the corresponding output $y = \pi(x; \theta) \in \Omega|_Y$, and updates its internal state, $\theta' = u(x; \theta) \in \Omega|_\Theta$. Right: We will also consider an idealized setting in which a self-predicting agent exerts full control over its experience by reading and writing to an internal boundary-space, $\Omega|_{b(\Theta)}$.
• Figure 4: Deep ReLU networks fail to sustain performance on the interactivity evaluation task. Interactivity-seeking simulates a big world and directly evaluates a learning algorithm's capability for continually adaptive behaviour. Right: The deep linear network sustains interactivity, whereas the deep ReLU collapses in performance. Middle: Each colour corresponds to one component of the action vector. The deep linear policy learns to produce actions following a non-stationary wave, which can be locally predicted by a static linear function and globally predicted by a dynamic linear function. Left: The deep ReLU policy fails to produce actions with any predictable structure, thus highlighting the challenge of interactivity-seeking as a continual learning problem.
• Figure 5: Deep linear networks are capable of sustaining higher interactivity with more computational resources. By increasing the width and depth of the deep linear network, we increase the network's capacity for continual adaptation, which allows it more quickly change its linear function approximator. Left: Increasing width marginally increases the sustained interactivity. Right: Increasing depth results in a large increase in interactivity, as well as more oscillatory behaviour.

Theorems & Definitions (18)

• Definition 1: Algorithmic Complexity
• Definition 2
• Proposition 1: Representing Turing machines
• Definition 3
• Definition 4
• Definition 5: Uniform Locality
• Definition 6
• Proposition 2: Automaton-Environment Relationship
• Proposition 3: Implicitly Constrained
• Definition 7: Interactivity