Beyond Attention: Toward Machines with Intrinsic Higher Mental States

TL;DR

The paper introduces Co^4, a cooperative, context-sensitive computation mechanism inspired by cellular neurobiology that preselects relevant information via triadic Q–K–V modulations before attention, achieving near-linear complexity in the number of input tokens. By embedding high-level perceptual processing and wakeful imagination into the representation level, Co^4 enables fast, parallel multi-perspective reasoning with substantially fewer layers and heads than standard Transformers. Across reinforcement learning, image classification, and synthetic NLP tasks, Co^4 demonstrates faster convergence and higher accuracy, while reducing gradient flow issues through an asynchronous modulation function. The findings suggest that biologically grounded, context-modulated processing can yield cognitively richer AI with improved efficiency, potentially scaling to large-scale models and informing future ethical and methodological directions.

Abstract

Attending to what is relevant is fundamental to both the mammalian brain and modern machine learning models such as Transformers. Yet, determining relevance remains a core challenge, traditionally offloaded to learning algorithms like backpropagation. Inspired by recent cellular neurobiological evidence linking neocortical pyramidal cells to distinct mental states, this work shows how models (e.g., Transformers) can emulate high-level perceptual processing and awake thought (imagination) states to pre-select relevant information before applying attention. Triadic neuronal-level modulation loops among questions ($Q$), clues (keys, $K$), and hypotheses (values, $V$) enable diverse, deep, parallel reasoning chains at the representation level and allow a rapid shift from initial biases to refined understanding. This leads to orders-of-magnitude faster learning with significantly reduced computational demand (e.g., fewer heads, layers, and tokens), at an approximate cost of $\mathcal{O}(N)$, where $N$ is the number of input tokens. Results span reinforcement learning (e.g., CarRacing in a high-dimensional visual setup), computer vision, and natural language question answering.

Figures (9)

• Figure 1: A pyramidal two-point neuron in the mammalian neocortex larkum1999newPhillips2024cellular integrates information at two functionally distinct sites. Its apical dendrites integrate CF at the contextual integration zone, while RF is integrated at the basal compartment, also known as the feedforward integration zone. The neuron triggers high-frequency firing (bursting) when RF and CF are matched in time, that is, when both the basal and apical zones are depolarized. This results in the amplification of coherent signals, enabling enhanced processing of contextually relevant information.
• Figure 2: An example of "thinking fast and slow" as discussed in kahneman2011thinking, illustrates how solving a riddle can involve combining rapid, intuitive processing in high-level perceptual state (fast thinking) with more deliberate, reflective refinement in the awake thought state (slow thinking).
• Figure 3: $Co^4$ architecture: $N$ denotes the number of input tokens, and each token has an embedding dimension of $E$. $Q_1$, $Q_2$,...,$Q_L$ represent the latent query tokens input to the associated Q-TPNs. $K_1$, $K_2$,...,$K_N$ represent the Key tokens input to the associated K-TPNs. $V_1$, $V_2$,...,$V_N$ represent the Value tokens input to the associated V-TPNs. This configuration forms part of the “seeing” state (i.e., sensory processing). In the “seeing as” state (i.e., perceptual and interpretive state), triadic modulation loops among questions ($Q$), clues (keys, $K$), and hypotheses (values, $V$) are executed through distal (D) and universal (U) contexts. Proximal (P) context represents normalization via information from neighboring neurons in the same population, including the prior information from the same neuron. The TPNs associated with $Q$, $K$, and $V$ are assumed to be analogous to three subtypes of pyramidal neurons, although their exact correspondence to neurobiologically distinguished subtypes is still under investigation. Through varying states of mind, high-level perceptual processing and wakeful thought, diverse, parallel reasoning chains are enabled. This mechanism incurs a computational cost of $\mathcal{O}(N \cdot L)$, where $L$ is a small fraction of the input length, making the overall cost approximately $\mathcal{O}(N)$. The triadic modulation loops, based on element-wise operations, add a nominal cost of $L \cdot N \cdot E$, which is significantly lower than that of the feedforward residual network used in standard Transformer blocks, a component $Co^4$ does not require. $Co^4$ can be viewed as a parallel, representation-level, silent yet deep form of Chain-of-Thought (CoT) reasoning wei2022chain (a quiet mind), enabling multi-perspective inference without requiring sequential token-level generation, much like the brain’s cortico-thalamic modulation aru2020cellularPhillips2024cellularstorm2024integrative.
• Figure 4: The TPN-inspired asynchronous MOD function (unnormalized) uses CF, denoted as $C$, to selectively amplify or attenuate the transmission of relevant and irrelevant RF, denoted as $R$, respectively. C serves as the driving force because the strong contextual signals from neighboring neurons can override even strong but noisy $R$ signals at the single-cell level. The combined perceptions of multiple neighboring neurons, each drawing from diverse and contextually relevant sources, carry more weight than the output of any individual neuron. This mechanism enhances cooperation between R and C for the task at hand. This behavior becomes particularly significant at higher levels of abstraction, where imagination begins to emerge and direct perception becomes less dominant.
• Figure 5: Vector field visualization and contour plots for deeper insight into the MOD function. The MOD function utilizes C to split R into coherent and incoherent R streams. The MOD function assigns greater weight to C; if C is high, and R is negative, the output of the function in eq (1) is positive (i.e. it makes f(C,R) have a different sign from R). Conversely, if C is low, R is diminished regardless of its own value.