Composing Recurrent Spiking Neural Networks using Locally-Recurrent Motifs and Risk-Mitigating Architectural Optimization

TL;DR

This paper tackles the challenge of principled, scalable design of recurrent spiking neural networks (RSNNs) by introducing Sparsely-Connected Recurrent Motif Layer (SC-ML), a motif-based building block that enables large-scale RSNNs with sparse inter-motif connectivity. It jointly develops Hybrid Risk-Mitigating Architectural Search (HRMAS), an alternating two-step optimization that first searches motif topology and weights and then applies intrinsic plasticity (IP) to stabilize the network during architectural evolution. The approach combines continuous relaxations of architectural choices with gradient-based optimization and a biologically inspired IP mechanism, achieving state-of-the-art results across four datasets (TI46-Alpha, N-TIDIGITS, DVS-Gesture, N-MNIST), including up to 3.38% performance gains over manually designed RSNNs at equivalent sizes. The work demonstrates that principled architectural optimization of RSNNs can yield robust, scalable neuromorphic models, with broad applicability and potential energy-efficiency benefits in neuromorphic hardware. ${SC-ML}$ and ${HRMAS}$ together provide a versatile framework for future brain-inspired recurrent architectures and their deployment on dedicated platforms.

Abstract

In neural circuits, recurrent connectivity plays a crucial role in network function and stability. However, existing recurrent spiking neural networks (RSNNs) are often constructed by random connections without optimization. While RSNNs can produce rich dynamics that are critical for memory formation and learning, systemic architectural optimization of RSNNs is still an open challenge. We aim to enable systematic design of large RSNNs via a new scalable RSNN architecture and automated architectural optimization. We compose RSNNs based on a layer architecture called Sparsely-Connected Recurrent Motif Layer (SC-ML) that consists of multiple small recurrent motifs wired together by sparse lateral connections. The small size of the motifs and sparse inter-motif connectivity leads to an RSNN architecture scalable to large network sizes. We further propose a method called Hybrid Risk-Mitigating Architectural Search (HRMAS) to systematically optimize the topology of the proposed recurrent motifs and SC-ML layer architecture. HRMAS is an alternating two-step optimization process by which we mitigate the risk of network instability and performance degradation caused by architectural change by introducing a novel biologically-inspired "self-repairing" mechanism through intrinsic plasticity. The intrinsic plasticity is introduced to the second step of each HRMAS iteration and acts as unsupervised fast self-adaptation to structural and synaptic weight modifications introduced by the first step during the RSNN architectural "evolution". To the best of the authors' knowledge, this is the first work that performs systematic architectural optimization of RSNNs. Using one speech and three neuromorphic datasets, we demonstrate the significant performance improvement brought by the proposed automated architecture optimization over existing manually-designed RSNNs.

Figures (6)

• Figure 1: Sparsely-Connected Recurrent Motif Layer.
• Figure 2: Architectural optimization in HRMAS.
• Figure 3: Evolution in neural development.
• Figure 4: Proposed HRMAS.
• Figure 5: SC-ML with relaxed architectural parameters.