Verti-Bench: A General and Scalable Off-Road Mobility Benchmark for Vertically Challenging Terrain

TL;DR

Off-road mobility benchmarks are needed to compare systems across diverse platforms and terrains, but existing work lacks standardized evaluation of unfolded vehicle-terrain interactions. The paper introduces Verti-Bench, a Chrono-based benchmark targeting vertically challenging terrain, featuring $100$ environments and $1000$ navigation tasks represented as $2.5$D elevation maps, with ten terrain semantics, random and expert datasets, and a gym-like RL interface. It provides a scalable suite of vehicle platforms and standardized metrics to quantify mobility performance, and evaluates ten mobility systems (classical, learning-based, and hybrid) to reveal strengths of 6-DoF kinodynamic planning in difficult terrain. Real-world validation on a physical 1/10th-scale testbed confirms simulation trends, underscoring the benchmark’s potential to drive data-driven advancement and future improvements in off-road mobility research.

Abstract

Recent advancement in off-road autonomy has shown promises in deploying autonomous mobile robots in outdoor off-road environments. Encouraging results have been reported from both simulated and real-world experiments. However, unlike evaluating off-road perception tasks on static datasets, benchmarking off-road mobility still faces significant challenges due to a variety of factors, including variations in vehicle platforms and terrain properties. Furthermore, different vehicle-terrain interactions need to be unfolded during mobility evaluation, which requires the mobility systems to interact with the environments instead of comparing against a pre-collected dataset. In this paper, we present Verti-Bench, a mobility benchmark that focuses on extremely rugged, vertically challenging off-road environments. 100 unique off-road environments and 1000 distinct navigation tasks with millions of off-road terrain properties, including a variety of geometry and semantics, rigid and deformable surfaces, and large natural obstacles, provide standardized and objective evaluation in high-fidelity multi-physics simulation. Verti-Bench is also scalable to various vehicle platforms with different scales and actuation mechanisms. We also provide datasets from expert demonstration, random exploration, failure cases (rolling over and getting stuck), as well as a gym-like interface for reinforcement learning. We use Verti-Bench to benchmark ten off-road mobility systems, present our findings, and identify future off-road mobility research directions.

Figures (6)

• Figure 0: Top: Low, Medium, and High Elevation Maps; Bottom: Elevation Histograms across Three Elevation Levels.
• Figure 1: Seven Rigid (percentage, mean and variance of friction coefficient, and texture) and Three Deformable (percentage, deformability, and texture) Terrain Semantics.
• Figure 2: Top: Sparse, Medium, and Dense Obstacles (black) and Global Paths (red) between Start and Goal (green). Bottom: Corresponding simulation scenario in Verti-Bench (elevation and semantics are removed for obstacle clarity).
• Figure 3: Verti-Bench Vehicles with Different Scale (1/10th, 1/6th, and full scale), Chassis (4-, 6-, and 8-wheeled and 2-tracked), Steering (pitman-arm, rack-and-pinion, toebar, bellcrank/rotary arm, and differential), and Tires (rigid and handling).
• Figure 4: Success Rate, Traversal Time, Roll, and Pitch with respect to Elevation Level (top), Terrain Semantics (middle), and Obstacle Density (bottom) of Ten Off-Road Mobility Systems on 1000 Navigation Tasks.