Robust Reinforcement Learning-Based Locomotion for Resource-Constrained Quadrupeds with Exteroceptive Sensing

TL;DR

This work tackles robust, real-time quadrupedal locomotion for small, resource-constrained robots on uneven terrain by integrating exteroceptive sensing with real-time elevation mapping and concurrent training of a policy and a state estimator. The system uses a minimal stereo plus ToF camera setup, GPU-accelerated elevation mapping, and an EKF that fuses estimator odometry with IMU and optionally VIO to provide robust perception and control. Key findings show the robot can traverse steps up to $17.5$ cm without failure and achieve an $80\%$ success rate on $22.5$ cm steps, while maintaining forward and yaw velocity tracking of up to $1.0$ m/s and $1.5$ rad/s, respectively, and enabling operation with or without VIO to save computation. The approach is validated through extensive real-world experiments, ablations, and mapping/odometry analyses, with code made open-source to facilitate adoption and further development.

Abstract

Compact quadrupedal robots are proving increasingly suitable for deployment in real-world scenarios. Their smaller size fosters easy integration into human environments. Nevertheless, real-time locomotion on uneven terrains remains challenging, particularly due to the high computational demands of terrain perception. This paper presents a robust reinforcement learning-based exteroceptive locomotion controller for resource-constrained small-scale quadrupeds in challenging terrains, which exploits real-time elevation mapping, supported by a careful depth sensor selection. We concurrently train both a policy and a state estimator, which together provide an odometry source for elevation mapping, optionally fused with visual-inertial odometry (VIO). We demonstrate the importance of positioning an additional time-of-flight sensor for maintaining robustness even without VIO, thus having the potential to free up computational resources. We experimentally demonstrate that the proposed controller can flawlessly traverse steps up to 17.5 cm in height and achieve an 80% success rate on 22.5 cm steps, both with and without VIO. The proposed controller also achieves accurate forward and yaw velocity tracking of up to 1.0 m/s and 1.5 rad/s respectively. We open-source our training code at github.com/ETH-PBL/elmap-rl-controller.

Figures (5)

• Figure 1: Top: Real-world stair climbing experiment with annotated sensors. Bottom: Visualization of the robot's state, showing filtered point clouds from the depth sensors, the elevation map, and the height samples passed to the policy.
• Figure 2: Deployed System overview: estimated odometry from the RL model is fused with IMU data and optionally VIO through an Extended Kalman Filter (EKF). The odometry along with the exteroceptive sensors are used to create an elevation map, out of which height samples are extracted and fed along with the proprioceptive sensors into the RL model, which outputs both actions and estimations. The Jetson Orin Nano processes the VIO and elevation mapping, while the Intel NUC handles model inference and EKF computation.
• Figure 3: Concurrent training framework: the policy is optimized with PPO, and the estimator is trained via supervised learning. The observation vector $\mathbf{o_t}$ is constructed from a history buffer. The actor receives a concatenation of observations and estimated variables $\mathbf{\tilde{x}_t}$ and outputs actions $\mathbf{a_t}$, while the critic receives the observations concatenated with privileged variables $\mathbf{x_t}$, which are also used to train the estimator. Rewards $\mathbf{r_t}$ are provided by the simulated environment.
• Figure 4: Comparison of total reward and linear velocity tracking reward across various models.
• Figure 5: Step traversal real-world experiment, where the success rate of various configurations is plotted against step height.