Neuronal Group Communication for Efficient Neural representation
Zhengqi Pei, Qingming Huang, Shuhui Wang
TL;DR
This paper introduces Neuronal Group Communication (NGC), a framework that reinterprets large neural networks as dynamical systems of interacting neuronal groups, with weights represented by low-rank interactions between embedding-like neuronal states $A$ and $B$ such that $W\approx AB^T$. It formalizes intra- and inter-group communication through a policy that yields a globally contractive, yet externally driven, dynamic system, and defines a Lyapunov-like neuronal stability score to quantify reasoning capability. The methodology includes initializing neuronal states via SVD, enforcing inter-group state sharing, and training only the neuronal states under a stability-regularized objective, enabling substantial compression with improved reasoning benchmarks on open-source LLMs. Empirically, NGC outperforms SVD-based and Basis Sharing baselines at similar compression ratios across multiple tasks, with ablations showing benefits from symmetric, learnable interaction metrics and higher effective latent dimensions. Beyond performance, NGC offers interpretability by exposing modular neuron groups and their communications, linking stability, geometry of state-space, and emergent reasoning to a principled dynamical framework with potential broad applicability to AI systems and cognitive-inspired architectures.
Abstract
The ever-increasing scale of modern neural networks has brought unprecedented performance alongside daunting challenges in efficiency and interpretability. This paper addresses the core question of how to build large neural systems that learn efficient, modular, and interpretable representations. We propose Neuronal Group Communication (NGC), a theory-driven framework that reimagines a neural network as a dynamical system of interacting neuronal groups rather than a monolithic collection of neural weights. Instead of treating each weight as an independent trainable parameter, NGC treats weights as transient interactions between embedding-like neuronal states, with neural computation unfolding through iterative communication among groups of neurons. This low-rank, modular representation yields compact models: groups of neurons exchange low-dimensional signals, enabling intra-group specialization and inter-group information sharing while dramatically reducing redundant parameters. By drawing on dynamical systems theory, we introduce a neuronal stability metric (analogous to Lyapunov stability) that quantifies the contraction of neuron activations toward stable patterns during sequence processing. Using this metric, we reveal that emergent reasoning capabilities correspond to an external driving force or ``potential'', which nudges the neural dynamics away from trivial trajectories while preserving stability. Empirically, we instantiate NGC in large language models (LLMs) and demonstrate improved performance on complex reasoning benchmarks under moderate compression. NGC consistently outperforms standard low-rank approximations and cross-layer basis-sharing methods at comparable compression rates. We conclude by discussing the broader implications of NGC, including how structured neuronal group dynamics might relate to generalization in high-dimensional learning systems.