GenTe: Generative Real-world Terrains for General Legged Robot Locomotion Control

TL;DR

GenTe presents a framework for generating realistic, physics-grounded terrains to train general legged locomotion policies. By coupling geometric height-map terrains with terramechanics-based physics and employing Vision-Language Models with function-calling, GenTe enables zero-shot terrain creation from text or images and a scalable curriculum for RL training. The work introduces a 100-terrain benchmark and demonstrates improved generalization and obstacle-avoidance能力 in bipedal robots, validated in simulation with PPO and a PD controller. This approach advances terrain-aware learning and provides open-source tools to accelerate research in robust legged locomotion for real-world, unstructured environments.

Abstract

Developing bipedal robots capable of traversing diverse real-world terrains presents a fundamental robotics challenge, as existing methods using predefined height maps and static environments fail to address the complexity of unstructured landscapes. To bridge this gap, we propose GenTe, a framework for generating physically realistic and adaptable terrains to train generalizable locomotion policies. GenTe constructs an atomic terrain library that includes both geometric and physical terrains, enabling curriculum training for reinforcement learning-based locomotion policies. By leveraging function-calling techniques and reasoning capabilities of Vision-Language Models (VLMs), GenTe generates complex, contextually relevant terrains from textual and graphical inputs. The framework introduces realistic force modeling for terrain interactions, capturing effects such as soil sinkage and hydrodynamic resistance. To the best of our knowledge, GenTe is the first framework that systemically generates simulation environments for legged robot locomotion control. Additionally, we introduce a benchmark of 100 generated terrains. Experiments demonstrate improved generalization and robustness in bipedal robot locomotion.

Figures (8)

• Figure 1: Terrains generated by GenTe with corresponding text/image prompts.
• Figure 2: Analysis of bipedal robots walking on wading terrains.
• Figure 3: Analysis of bipedal robots walking on deformable terrains.
• Figure 4: Structure of GenTe. The proposed pipeline for terrain generation and policy training in a simulated environment for bipedal robot locomotion control. During policy training, the robots are trained on individual basic terrains in a curriculum-based progression. In the inference phase, the task specification is provided either as a text description or an image. The LLM then calls functions from the terrain curriculum to apply relevant geometry and physical properties, creating a targeted, real-world-inspired terrain. Examples of basic terrains and generated terrains are shown on the right.
• Figure 5: Averaged training reward with variance intervals over three trails. The solid blue line represents the mean reward over training iterations, while the shaded region indicates the variance, demonstrating the stability and improvement of the agent’s performance.