Modern Neuromorphic AI: From Intra-Token to Inter-Token Processing

TL;DR

The paper argues that modern AI is increasingly embracing neuromorphic principles by combining intra-token processing (channel-wise mixing within tokens) with inter-token processing (cross-token interactions via state dynamics and sparse attention). It formalizes Neuromorphic Processing Units (NPUs) and Neuromorphic Processing Elements (NPEs), detailing discrete activations and stateful dynamics, and surveys intra-token processing along a virtual time axis as well as inter-token processing via state-space dynamics and self-attention. It discusses practical instantiations such as rate/temporal encodings, RWKV-inspired inter-token dynamics, and spiking attention methods, while addressing training via teacher forcing with surrogate gradients and local, reinforcement-like updates. The work highlights energy-accuracy trade-offs, scalable parallelization opportunities, and hardware-software co-design as essential for edge-deployed neuromorphic AI, outlining open research directions in efficient language/multimodal models and zeroth-order training methods. Overall, it provides a unifying framework to understand how neuromorphic concepts can be integrated into both intra-token and inter-token computations to improve the intelligence-to-joule ratio in next-generation AI systems.

Abstract

The rapid growth of artificial intelligence (AI) has brought novel data processing and generative capabilities but also escalating energy requirements. This challenge motivates renewed interest in neuromorphic computing principles, which promise brain-like efficiency through discrete and sparse activations, recurrent dynamics, and non-linear feedback. In fact, modern AI architectures increasingly embody neuromorphic principles through heavily quantized activations, state-space dynamics, and sparse attention mechanisms. This paper elaborates on the connections between neuromorphic models, state-space models, and transformer architectures through the lens of the distinction between intra-token processing and inter-token processing. Most early work on neuromorphic AI was based on spiking neural networks (SNNs) for intra-token processing, i.e., for transformations involving multiple channels, or features, of the same vector input, such as the pixels of an image. In contrast, more recent research has explored how neuromorphic principles can be leveraged to design efficient inter-token processing methods, which selectively combine different information elements depending on their contextual relevance. Implementing associative memorization mechanisms, these approaches leverage state-space dynamics or sparse self-attention. Along with a systematic presentation of modern neuromorphic AI models through the lens of intra-token and inter-token processing, training methodologies for neuromorphic AI models are also reviewed. These range from surrogate gradients leveraging parallel convolutional processing to local learning rules based on reinforcement learning mechanisms.

Figures (8)

• Figure 1: Modern AI systems pre-process text -- and similarly other types of data -- via tokenization and embedding. These steps produce a matrix $\bm E$ with $N$ columns (here $N=5$), each corresponding to a token embedding containing $D$ channels.
• Figure 2: Modern AI systems typically alternate between intra-token processing steps, such as linear projections and feedforward neural networks (also known as multi-layer perceptrons), and inter-token processing, such as attention and state-based dynamics. The figure represents a single layer of computation. Note that in this simplified picture, important blocks such as concatenation, normalization, multi-head processing, and residual connections are not represented.
• Figure 3: Intra-token processing operates on each token $n$ independently, combining information across channels $d=1,...,D$ within a single token via a function of the form (\ref{['eq:intra']}). The output token has a generally different dimension $D'$.
• Figure 4: Neuromorphic computing can enhance the energy efficiency of both intra-token processing and inter-token processing: (top) In most early work, the focus was on enhancing the energy efficiency of intra-token processing steps by operating over a virtual time axis via sparse spiking signals; (bottom) More recently, there has been interest in porting the benefits of dynamic sparsity via discrete activations, which is typical of neuromorphic processors, also to inter-token processing.
• Figure 5: Inter-token processing operates on each channel $d$ independently, mixing information from different tokens $n=1,...,N$ via a function of the form (\ref{['eq:inter']}). When masking is applied, only some input tokens can contribute to each $n$-th output token (the figure illustrates the case of an autoregressive mask, with inactive dependencies in gray). The number of output tokens $N'$ can exceed $N$, as is the case in generative language models (illustrated with suspension points).