The Thousand Brains Project: A New Paradigm for Sensorimotor Intelligence

TL;DR

The paper tackles the challenge of general, real-world sensorimotor intelligence by proposing the Thousand Brains Project, which reimagines AI as a cortex-inspired, modular, sensorimotor system. It introduces Monty, a first implementation where learning modules (cortical-column–like units) operate with structured graphs, reference frames, and a Cortical Messaging Protocol to communicate across modalities and hierarchy. Key contributions include the CMP framework, graph-based object models with buffers and long-term memory, evidence-driven learning, and heterarchical LM connectivity enabling rapid, multimodal inference and goal-directed action. The work aims to deliver a scalable, embodied AI platform capable of rapid continual learning and generalization, offering a complementary direction to deep learning with potential broad impact on robotics and intelligent systems.

Abstract

Artificial intelligence has advanced rapidly in the last decade, driven primarily by progress in the scale of deep-learning systems. Despite these advances, the creation of intelligent systems that can operate effectively in diverse, real-world environments remains a significant challenge. In this white paper, we outline the Thousand Brains Project, an ongoing research effort to develop an alternative, complementary form of AI, derived from the operating principles of the neocortex. We present an early version of a thousand-brains system, a sensorimotor agent that is uniquely suited to quickly learn a wide range of tasks and eventually implement any capabilities the human neocortex has. Core to its design is the use of a repeating computational unit, the learning module, modeled on the cortical columns found in mammalian brains. Each learning module operates as a semi-independent unit that can model entire objects, represents information through spatially structured reference frames, and both estimates and is able to effect movement in the world. Learning is a quick, associative process, similar to Hebbian learning in the brain, and leverages inductive biases around the spatial structure of the world to enable rapid and continual learning. Multiple learning modules can interact with one another both hierarchically and non-hierarchically via a "cortical messaging protocol" (CMP), creating more abstract representations and supporting multimodal integration. We outline the key principles motivating the design of thousand-brains systems and provide details about the implementation of Monty, our first instantiation of such a system. Code can be found at https://github.com/thousandbrainsproject/tbp.monty, along with more detailed documentation at https://thousandbrainsproject.readme.io/.

Figures (25)

• Figure 1: Sensor modules receive and process the raw sensory and motor input. This is then communicated via a common messaging protocol to a learning module which uses this information to learn and recognize models of anything in the environment.
• Figure 2: Learning modules learn structured models through sensorimotor interaction, using reference frames. They model how incoming features are arranged relative to each other in space.
• Figure 3: Conceptual sketch of how the learning module could be implementing possible mechanisms of cortical columns. The figure on the right represents three cortical columns, including cellular layers. The internal structure of a learning module can be mapped onto these layers.
• Figure 4: By using a common messaging protocol between sensor modules and learning modules, the system can easily be scaled in multiple dimensions. This provides a straightforward way for dealing with multiple sensory modalities. Using multiple learning modules next to each other can improve robustness and speed through votes between them. Additionally, stacking learning modules on top of each other allows for more complex, hierarchical processing of inputs and modeling compositional objects.
• Figure 5: High-level overview of the architecture with all the main conceptual components mirroring Figure \ref{['fig:Scaling']} applied to a concrete example. Blue lines indicate the feed-forward flow of information up the hierarchy. Purple lines show top-down connections, biasing the lower-level LMs. Green lines show lateral voting connections. Pink lines show the communication of goal states, which eventually translate into motor commands in the motor system. Every LM has a direct motor output. Information communicated along solid lines follows the CMP (contains features and pose). Discontinuations in the diagram are marked with dots on line ends. Dashed lines are the interface of the system with the world and do not need to follow the CMP. Blue dashed lines communicate raw sensory input from sensors. Pink dashed lines communicate motor commands to the actuators. The large, semi-transparent blue arrow is an example of a connection carrying sensory outputs from a larger receptive field directly to a higher-level LM.