Continuous Thought Machines

TL;DR

The Continuous Thought Machine introduces neuron-level temporal processing and neural synchronization as core computational primitives, reintroducing neural timing into artificial intelligence. By employing an internal sequence dimension of ticks, privately parameterized neuron-level models, and synchronization-based latent representations, CTM enables adaptive compute and rich internal dynamics across diverse tasks such as 2D mazes, ImageNet-1K, and parity algorithms. Its results demonstrate native adaptive processing, interpretable sequential strategies, and emergent dynamics within a biologically inspired yet differentiable framework. While not optimized for state-of-the-art accuracy, CTM offers a compelling pathway toward more biologically plausible and temporally rich AI, with broad potential for language, memory, and multi-modal reasoning.»

Abstract

Biological brains demonstrate complex neural activity, where neural dynamics are critical to how brains process information. Most artificial neural networks ignore the complexity of individual neurons. We challenge that paradigm. By incorporating neuron-level processing and synchronization, we reintroduce neural timing as a foundational element. We present the Continuous Thought Machine (CTM), a model designed to leverage neural dynamics as its core representation. The CTM has two innovations: (1) neuron-level temporal processing, where each neuron uses unique weight parameters to process incoming histories; and (2) neural synchronization as a latent representation. The CTM aims to strike a balance between neuron abstractions and biological realism. It operates at a level of abstraction that effectively captures essential temporal dynamics while remaining computationally tractable. We demonstrate the CTM's performance and versatility across a range of tasks, including solving 2D mazes, ImageNet-1K classification, parity computation, and more. Beyond displaying rich internal representations and offering a natural avenue for interpretation owing to its internal process, the CTM is able to perform tasks that require complex sequential reasoning. The CTM can also leverage adaptive compute, where it can stop earlier for simpler tasks, or keep computing when faced with more challenging instances. The goal of this work is to share the CTM and its associated innovations, rather than pushing for new state-of-the-art results. To that end, we believe the CTM represents a significant step toward developing more biologically plausible and powerful artificial intelligence systems. We provide an accompanying interactive online demonstration at https://pub.sakana.ai/ctm/ and an extended technical report at https://pub.sakana.ai/ctm/paper .

Figures (35)

• Figure 1: Solving 100 steps down $39 \times 39$ mazes. (a, b) Observing using attention (no positional encoding, weights overlaid), imagining a route (arrows) from red to green pixels, (b) attending beyond 100 steps, and (c) generalizing to $99 \times 99$ via sequential re-applications of the same model.
• Figure 2: ImageNet-1K demonstration. (a) Complex neural dynamics whose synchronization are the representation with which the CTM observes and predict. (b) CTM's attention process, showing all 16 attention heads (left) and average thereof (middle). Arrows trace the average weighting over internal ticks, exemplifying a complex path that emerges without any training signal. We discuss more interesting emergent properties of the CTM in \ref{['app:emergent']}. Video demonstrations are https://pub.sakana.ai/ctm/#imagenet-demos.
• Figure 3: CTM architecture overview. Key components include: Synapse model generating pre-activations from prior post-activations $\mathbf{z}^t$ and attention output $\mathbf{o}^t$. History of pre-activations $\mathbf{A}^t$. Neuron-level models (NLMs) processing $\mathbf{A}_d^t$ to produce post-activations $\mathbf{z}^{t+1}_d$. History of post-activations $\mathbf{Z}^t$. Neural synchronization matrix $\mathbf{S}^t$ computed from $\mathbf{Z}^t$. Selected neuron pairs from $\mathbf{S}^t$ form latent representations used for outputs $\mathbf{y}^t$ and attention queries $\mathbf{q}^t$. Attention output $\mathbf{o}^t$ is concatenated with $\mathbf{z}^{t+1}$ for the next internal tick. Owing to the inherent difficulty in visualizing a dynamic, time-based architecture, we include the supplementary video 'arch.mp4' (hosted https://pub.sakana.ai/ctm/#method too) that visualizes functional data flow.
• Figure 4: CTM versus baselines on 2D mazes. The CTM demonstrates superior trainability compared to baselines, yielding higher accuracy for longer paths. Using iterative re-applications, we show in (b) that the CTM can generalise to longer paths and bigger mazes. See \ref{['sec:appendix-mazes-results']} for loss curves.
• Figure 5: ImageNet-1K results: (a) Native adaptive compute potential based on a 0.8 certainty threshold, showing performance expected at each internal tick. (b) Excellent model calibration when averaging probabilities up to each tick shown. See Appendix \ref{['sec:appendix-imagenet-prediction']} for further analysis.