Event-driven Robust Fitting on Neuromorphic Hardware

TL;DR

This work tackles the rising energy cost of robust geometric fitting in computer vision by introducing NeuroRF, an event-driven spiking neural network implemented on Intel Loihi 2. NeuroRF uses novel lifted gradient formulations and an event-driven architecture to perform random sampling, model hypothesis generation, and model verification entirely within neuromorphic hardware, with careful mitigations for limited precision and instructions. CPU simulations show competitive accuracy against established robust-fitting methods, while on-chip measurements demonstrate substantial energy savings (approximately 15% of CPU energy) at equivalent accuracy, albeit with longer runtimes. The approach is demonstrated on affine image registration, achieving competitive AUC performance against RANSAC-family solvers and highlighting Neuromorphic robustness and energy efficiency as a practical route for low-power, real-time vision pipelines.

Abstract

Robust fitting of geometric models is a fundamental task in many computer vision pipelines. Numerous innovations have been produced on the topic, from improving the efficiency and accuracy of random sampling heuristics to generating novel theoretical insights that underpin new approaches with mathematical guarantees. However, one aspect of robust fitting that has received little attention is energy efficiency. This performance metric has become critical as high energy consumption is a growing concern for AI adoption. In this paper, we explore energy-efficient robust fitting via the neuromorphic computing paradigm. Specifically, we designed a novel spiking neural network for robust fitting on real neuromorphic hardware, the Intel Loihi 2. Enabling this are novel event-driven formulations of model estimation that allow robust fitting to be implemented in the unique architecture of Loihi 2, and algorithmic strategies to alleviate the current limited precision and instruction set of the hardware. Results show that our neuromorphic robust fitting consumes only a fraction (15%) of the energy required to run the established robust fitting algorithm on a standard CPU to equivalent accuracy.

Figures (8)

• Figure 1: The top diagram shows a high-level schematic of a neuro core in Loihi 2. Note that Neuron block is programmable. The bottom diagram shows our technique to emulate matrix-matrix multiplication and how it is mapped onto the neuro core. See Fig. AA Supplementary for more details.
• Figure 2: The proposed NeuroRF SNN.
• Figure 3: Normalized Euclidean distance $(\%)$ across various levels of difficulty. Results were averaged over 10 trials for each method.
• Figure 4: Performance on neuromorphic hardware. (a) Normalized Euclidean distance $(\%)$, (b) average dynamic energy consumption and (c) average runtime of NeuroRF-Loihi and RS-CPU on synthetic line fitting instances with $N = 20$ points, plotted against outlier rate.
• Figure 5: Green and red lines represent inliers and outliers found by NeuroRS-CPU on two affine image registration instances.