Resonant Sparse Geometry Networks

TL;DR

This paper addresses the quadratic bottleneck of dense self-attention by proposing Resonant Sparse Geometry Networks (RSGN), which embed computational nodes in learned hyperbolic space and use input-dependent ignition to create sparse, hierarchical connectivity. It introduces a two-timescale learning framework that combines fast gradient-based activation routing with slow Hebbian structural plasticity, including synaptic pruning and sprouting, all under a global reward signal. Theoretical analysis establishes sub-quadratic complexity $O(K \cdot N \cdot d_h^2)$ under sparse activation and locality, and empirical results demonstrate strong parameter efficiency: 96.5% accuracy on long-range dependency tasks with ~40k parameters, and 23.8% on hierarchical classification with ~41k parameters, competitive with Transformer baselines that use orders of magnitude more parameters. The work suggests brain-inspired sparse, geometrically organized computation can achieve practical efficiency while preserving expressive power, with implications for neuromorphic hardware and scalable AI systems.

Abstract

We introduce Resonant Sparse Geometry Networks (RSGN), a brain-inspired architecture with self-organizing sparse hierarchical input-dependent connectivity. Unlike Transformer architectures that employ dense attention mechanisms with O(n^2) computational complexity, RSGN embeds computational nodes in learned hyperbolic space where connection strength decays with geodesic distance, achieving dynamic sparsity that adapts to each input. The architecture operates on two distinct timescales: fast differentiable activation propagation optimized through gradient descent, and slow Hebbian-inspired structural learning for connectivity adaptation through local correlation rules. We provide rigorous mathematical analysis demonstrating that RSGN achieves O(n*k) computational complexity, where k << n represents the average active neighborhood size. Experimental evaluation on hierarchical classification and long-range dependency tasks demonstrates that RSGN achieves 96.5% accuracy on long-range dependency tasks while using approximately 15x fewer parameters than standard Transformers. On challenging hierarchical classification with 20 classes, RSGN achieves 23.8% accuracy (compared to 5% random baseline) with only 41,672 parameters, nearly 10x fewer than the Transformer baselines which require 403,348 parameters to achieve 30.1% accuracy. Our ablation studies confirm the contribution of each architectural component, with Hebbian learning providing consistent improvements. These results suggest that brain-inspired principles of sparse, geometrically-organized computation offer a promising direction toward more efficient and biologically plausible neural architectures.

Figures (8)

• Figure 1: Bio-inspired principles underlying RSGN.(A) Resonant Sparse Geometry Network (RSGN) architecture illustrating the four key principles: (1) input-dependent routing where input tokens create spark points that ignite nearby nodes; (2) hierarchical sparse connectivity with distance-based connection strength; (3) two-timescale learning combining fast gradient-based activation updates with slow Hebbian structural plasticity; and (4) hyperbolic geometry embedding with soft thresholds and local inhibition. The brain illustration shows analogous biological mechanisms including attentional networks and striatal reward modulation. (B) Bio-inspired Modular Representation comparing cortical organization (left) with RSGN implementation (right). The cortical hierarchy shows sensory input propagating from primary sensory cortex through association areas to higher cognitive regions, with Hebbian plasticity ("neurons that fire together, wire together"). The corresponding RSGN diagram shows resonant signal propagation through the Poincaré ball across iterative steps, with the ignition module processing input sequences and producing classifier output through local inhibition and soft threshold operations.
• Figure 2: RSGN architecture overview. Input tokens are embedded and create spark points in the hyperbolic embedding space (Poincaré ball), where distance-based connectivity determines connection strength. Only $\sim$2% of nodes activate (sparse activation), with local inhibition implementing winner-take-more dynamics. Activations propagate iteratively through $T$ steps, followed by soft threshold activation, layer normalization, and readout. The two-timescale learning system combines fast gradient descent with differentiable relaxation and slow Hebbian plasticity for structural adaptation, both modulated by a global reward signal. The inset shows the Poincaré disk geometry where nodes closer to each other have stronger connections, naturally embedding tree-like hierarchies.
• Figure 3: Accuracy comparison on hierarchical classification task (20 classes). RSGN achieves 23.8% accuracy with only 41,672 parameters, compared to Transformer's 30.1% with 403,348 parameters. The random baseline for 20 classes is 5%, meaning RSGN achieves nearly 5$\times$ better than random with 10$\times$ fewer parameters than Transformer.
• Figure 4: Accuracy on long-range dependency task (sequence length 128, 10 classes). RSGN achieves 96.5% accuracy with 40,382 parameters, compared to Transformer and LSTM achieving 100% with approximately 15$\times$ more parameters. The strong performance demonstrates RSGN's ability to capture long-range dependencies despite using significantly fewer parameters.
• Figure 5: Ablation study results showing the contribution of each RSGN component. The relatively stable performance across configurations suggests robustness to hyperparameter choices, with Hebbian learning providing consistent benefits.

Theorems & Definitions (19)

• Definition 1: RSGN Node
• Definition 2: Poincaré Ball
• Remark 1: Geometric Intuition
• Definition 3: Connection Strength
• Definition 4: Ignition Function
• Remark 2: Biological Analogy
• Definition 5: Soft Threshold Function
• Definition 6: Propagation Dynamics
• Definition 7: Local Inhibition
• Definition 8: Output Readout