Tiny Reinforcement Learning for Quadruped Locomotion using Decision Transformers

TL;DR

This work tackles deploying imitation-learning–driven quadruped locomotion on ultra-resource robots by recasting the problem as conditional sequence modeling with a Decision Transformer and augmenting expert demonstrations with a platform-specific reward. It then compresses the resulting model using quantization and pruning to fit limited-memory hardware, demonstrating natural-looking gaits for the Bittle robot in simulation and showing that $4$-bit quantization can preserve performance while yielding substantial size reductions. The approach provides a practical path toward tinyML deployment of learning-based locomotion, balancing policy quality with hardware constraints and guiding future enhancements in reward design and transformer compression. Overall, the paper advances the feasibility of real-time, low-cost quadruped control on resource-constrained platforms with actionable compression strategies and empirical validation in simulation.

Abstract

Resource-constrained robotic platforms are particularly useful for tasks that require low-cost hardware alternatives due to the risk of losing the robot, like in search-and-rescue applications, or the need for a large number of devices, like in swarm robotics. For this reason, it is crucial to find mechanisms for adapting reinforcement learning techniques to the constraints imposed by lower computational power and smaller memory capacities of these ultra low-cost robotic platforms. We try to address this need by proposing a method for making imitation learning deployable onto resource-constrained robotic platforms. Here we cast the imitation learning problem as a conditional sequence modeling task and we train a decision transformer using expert demonstrations augmented with a custom reward. Then, we compress the resulting generative model using software optimization schemes, including quantization and pruning. We test our method in simulation using Isaac Gym, a realistic physics simulation environment designed for reinforcement learning. We empirically demonstrate that our method achieves natural looking gaits for Bittle, a resource-constrained quadruped robot. We also run multiple simulations to show the effects of pruning and quantization on the performance of the model. Our results show that quantization (down to 4 bits) and pruning reduce model size by around 30\% while maintaining a competitive reward, making the model deployable in a resource-constrained system.

Figures (4)

• Figure 1: Example gait for Bittle in simulated Isaac Gym environment
• Figure 2: Statistics of the reward obtained after quantization for different numbers of bits, based on 250 Monte-Carlo points.
• Figure 3: Statistics of the reward obtained for different combinations of quantization and pruning strategies based on 250 Monte-Carlo points.
• Figure 4: Normalized model size reduction across the different compression techniques applied.