Neuromorphic Attitude Estimation and Control

TL;DR

This work tackles the energy- and latency-constrained problem of quadrotor autonomy by proposing the first fully neuromorphic end-to-end attitude estimation and control pipeline implemented on a tiny drone. It trains two modular spiking neural networks—a state estimator and a controller—via imitation learning, employs time-shifted targets and data augmentation to mitigate delay and reality gap, and deploys the system on a Crazyflie with a Teensy co-processor at 500 Hz. The results show responsive attitude tracking with RMSE around $3.0$ degrees, competitive with a conventional flight stack, and reveal design strategies such as integrator neurons and network merging that stabilize learning and inference. The findings indicate that end-to-end neuromorphic autopilots are feasible and could unlock substantial energy efficiency and low-latency performance, especially when implemented on dedicated neuromorphic hardware.

Abstract

The real-world application of small drones is mostly hampered by energy limitations. Neuromorphic computing promises extremely energy-efficient AI for autonomous flight but is still challenging to train and deploy on real robots. To reap the maximal benefits from neuromorphic computing, it is necessary to perform all autonomy functions end-to-end on a single neuromorphic chip, from low-level attitude control to high-level navigation. This research presents the first neuromorphic control system using a spiking neural network (SNN) to effectively map a drone's raw sensory input directly to motor commands. We apply this method to low-level attitude estimation and control for a quadrotor, deploying the SNN on a tiny Crazyflie. We propose a modular SNN, separately training and then merging estimation and control sub-networks. The SNN is trained with imitation learning, using a flight dataset of sensory-motor pairs. Post-training, the network is deployed on the Crazyflie, issuing control commands from sensor inputs at 500Hz. Furthermore, for the training procedure we augmented training data by flying a controller with additional excitation and time-shifting the target data to enhance the predictive capabilities of the SNN. On the real drone, the perception-to-control SNN tracks attitude commands with an average error of 3.0 degrees, compared to 2.7 degrees for the regular flight stack. We also show the benefits of the proposed learning modifications for reducing the average tracking error and reducing oscillations. Our work shows the feasibility of performing neuromorphic end-to-end control, laying the basis for highly energy-efficient and low-latency neuromorphic autopilots.

Figures (5)

• Figure 1: We present an approach to training a spiking neural network for end-to-end attitude estimation and control of tiny drones (deployed on a Crazyflie, top). The network is a merging of a 2-layer attitude estimation sub-network with recurrency and a 1-layer recurrent attitude control network (bottom). The network exhibits a spiking activity of $15\%$, which is promising in terms of energy efficiency for future implementation on a neuromorphic processor. The network currently runs at 500Hz on a Teensy microcontroller.
• Figure 2: Pearson Correlation between the output of the trained SNN and the regular PID output for different time shifts $d$. The bottom graph shows the output of the network for time shifts $d=0$, $d=6$ and $d=12$ compared to the target, further demonstrating that a delay is present in the network.
• Figure 3: Training loss curves comparing fixed versus free neuron leak and threshold parameters. The proposed approach of fixing neuron parameters leads to stable convergence during training. Allowing these parameters to remain free results in training becoming trapped in local minima.
• Figure 4: Position step responses of the SNN system (top) and the regular PID flight stack (bottom) for 10 individual test runs. The SNN can accurately track the attitude references as given by the outer-loop position controller and maintain a stable flight path.
• Figure 5: Attitude step responses of A) the fully-trained SNN system, B) the SNN trained with augmentation, C) the SNN trained with time-shifted data and D) the regular PID flight stack. The images on top show the Crazyflie during the different maneuvers.