ES-Parkour: Advanced Robot Parkour with Bio-inspired Event Camera and Spiking Neural Network

TL;DR

This work tackles energy-efficient perception and control for quadruped parkour under challenging lighting by combining event cameras with Spiking Neural Networks (SNNs) through an ANN-to-SNN distillation workflow. The authors develop a simulation-based pipeline in IsaacGym that converts brightness changes into event streams and trains an ANN teacher before distilling its behavior into an energy-efficient SNN, using a curriculum across terrains and privileged information. Theoretical energy modeling and extensive simulations show substantial improvements, including energy reductions up to $88.3$% and total ES-Parkour energy consuming only $11.7$% of the ANN baseline. This approach advances robust perception and fast, low-power control for real-world, computation-constrained robotics, paving the way for deployment of brain-inspired sensing and processing in demanding environments.

Abstract

In recent years, quadruped robotics has advanced significantly, particularly in perception and motion control via reinforcement learning, enabling complex motions in challenging environments. Visual sensors like depth cameras enhance stability and robustness but face limitations, such as low operating frequencies relative to joint control and sensitivity to lighting, which hinder outdoor deployment. Additionally, deep neural networks in sensor and control systems increase computational demands. To address these issues, we introduce spiking neural networks (SNNs) and event cameras to perform a challenging quadruped parkour task. Event cameras capture dynamic visual data, while SNNs efficiently process spike sequences, mimicking biological perception. Experimental results demonstrate that this approach significantly outperforms traditional models, achieving excellent parkour performance with just 11.7% of the energy consumption of an artificial neural network (ANN)-based model, yielding an 88.3% energy reduction. By integrating event cameras with SNNs, our work advances robotic reinforcement learning and opens new possibilities for applications in demanding environments.

Figures (5)

• Figure 1: Demonstration of a quadruped robot performing parkour using spiking neural network under extreme lighting conditions. The robot processes event images in real-time, where the red part and blue part denote the positive event and negative event, respectively.
• Figure 2: Pipeline of our bio-inspired reinforcement learning system. Different from the previous standard vision-based robot system, our bio-inspired system is equipped with an event camera to capture event data from diverse scenes. The event is then processed by the spiking neural network which in turn dictates the robot's actions to the environment. The adoption of this brain-inspired approach yields three significant advantages: (1) enhanced stability in motion-intensive scenarios is achieved through the superior temporal resolution of the event data. (2) the system's resilience in fluctuating lighting conditions is ensured by the event camera's high dynamic range. (3) the inherently low energy consumption of the SNN contributes to the system's overall efficiency.
• Figure 3: Pipeline of our ES-Parkour ANN-to-SNN distilling process. Through the distillation process, the extreme parkour capabilities of the ANN are transferred to an SNN, which receives input from an event camera. In the warm-up phase, minimizing the Mean Squared Error (MSE) loss between the outputs of the teacher (ANN) and the student (SNN) networks ensures the student network can closely replicate the teacher network's outputs. Following the warm-up phase, the student network demonstrates basic movement capabilities but encounters challenges with complex terrains. Further interaction and optimization of the student network enhance its performance on complex terrains, closely aligning it with the teacher's performance.
• Figure 4: Overview of the event simulation process. Each depth image can be converted into its corresponding event with the optical flow and image gradient.
• Figure 5: We evaluate our SNN strategy across four different scenarios. The figure shows the shapes related to each. The top row indicates the type of terrain, while the bottom row displays the success rate for each situation.