One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion

TL;DR

The paper tackles the challenge of learning a single locomotion policy that can control diverse legged robot morphologies. It introduces URMA, a morphology-agnostic encoder-decoder architecture with an attention-based joint/feet description routing and a universal morphology decoder to produce actions for any robot morphology. Through extensive simulation across 16 robots and zero-shot real-world transfers to several quadrupeds, URMA demonstrates robust, transferable locomotion and outperforms morphology-specific baselines. The work also provides theoretical insights into multi-task risk bounds for shared representations and offers an open-source framework that can serve as a foundation for locomotion foundation models and broader control tasks.

Abstract

Deep Reinforcement Learning techniques are achieving state-of-the-art results in robust legged locomotion. While there exists a wide variety of legged platforms such as quadruped, humanoids, and hexapods, the field is still missing a single learning framework that can control all these different embodiments easily and effectively and possibly transfer, zero or few-shot, to unseen robot embodiments. We introduce URMA, the Unified Robot Morphology Architecture, to close this gap. Our framework brings the end-to-end Multi-Task Reinforcement Learning approach to the realm of legged robots, enabling the learned policy to control any type of robot morphology. The key idea of our method is to allow the network to learn an abstract locomotion controller that can be seamlessly shared between embodiments thanks to our morphology-agnostic encoders and decoders. This flexible architecture can be seen as a potential first step in building a foundation model for legged robot locomotion. Our experiments show that URMA can learn a locomotion policy on multiple embodiments that can be easily transferred to unseen robot platforms in simulation and the real world.

Figures (13)

• Figure 1: Top -- We train a single locomotion policy for multiple robot embodiments in simulation. Bottom -- We can transfer and deploy the policy on three real-world platforms by randomizing the embodiments and environment dynamics during training.
• Figure 2: Overview of URMA. Left -- Joint observations and descriptions are encoded and combined into a single joint latent vector through an attention head. Bottom center -- Feet observations and descriptions are encoded in the same way. Top center -- Joint latent, feet latent, and general observations are fed through the core network to get the action latent vector. Right -- The universal morphology decoder encodes the joint descriptions and pairs them with the action latent vector and the single joint latent vector to produce the action mean and standard deviation for the final action.
• Figure 3: Top left -- Average return of the three architectures during training on all 16 robots compared to the single-robot training setting. Top right -- Zero-shot transfer to the Unitree A1 while training on the other 15 robots. Bottom left -- Zero-shot transfer to the MAB Robotics Silver Badger while training on the other 15 robots and fine-tuning on only the Silver Badger afterward. Bottom right -- Zero-shot evaluation on all 16 robots while removing the feet observations.
• Figure 4: Ablation on the LayerNorm layers in the architecture.
• Figure 5: Ablation on the usage of two separate or one single joint description encoder in the joint encoder and the universal decoder.

Theorems & Definitions (1)

• Theorem 1