YOPO-Rally: A Sim-to-Real Single-Stage Planner for Off-Road Terrain

TL;DR

The paper tackles the challenge of off-road autonomous navigation in forested environments by introducing a zero-shot sim-to-real end-to-end planner, YOPO-Rally, trained entirely in the YOPO-Sim simulator and deployed with an MPC controller. It fuses Terrain Traversability Analysis and pathfinding into a single neural network that maps depth imagery, vehicle velocity, and a goal vector to multiple trajectory candidates represented as non-uniform cubic Hermite curves, along with associated costs. Training relies on behavior cloning from expert trajectories generated via TTA optimization in simulation, enabling direct real-world deployment without fine-tuning. Across extensive simulated and real-world experiments, the approach demonstrates fast planning, robust trajectory quality, and effective sim-to-real transfer, outperforming a state-of-the-art baseline in planning speed and safety while operating under constrained hardware on real platforms.

Abstract

Off-road navigation remains challenging for autonomous robots due to the harsh terrain and clustered obstacles. In this letter, we extend the YOPO (You Only Plan Once) end-to-end navigation framework to off-road environments, explicitly focusing on forest terrains, consisting of a high-performance, multi-sensor supported off-road simulator YOPO-Sim, a zero-shot transfer sim-to-real planner YOPO-Rally, and an MPC controller. Built on the Unity engine, the simulator can generate randomized forest environments and export depth images and point cloud maps for expert demonstrations, providing competitive performance with mainstream simulators. Terrain Traversability Analysis (TTA) processes cost maps, generating expert trajectories represented as non-uniform cubic Hermite curves. The planner integrates TTA and the pathfinding into a single neural network that inputs the depth image, current velocity, and the goal vector, and outputs multiple trajectory candidates with costs. The planner is trained by behavior cloning in the simulator and deployed directly into the real-world without fine-tuning. Finally, a series of simulated and real-world experiments is conducted to validate the performance of the proposed framework.

Figures (8)

• Figure 1: Visualization of the primitives and the predicted trajectories. The depth image projected to the far plane is divided into $M_\theta \times 1$ grids, corresponding to the primitive anchors $\bm{p}_i$ in the body frame. The end-state norm $p_n$ and angle $p_\theta$ predicted are the offsets to the primitive anchor $\bm{p}_i$.
• Figure 2: System overview. The planner inputs the depth image and the state consisting of velocity and goal vector, and predicts the end-state offsets and costs. Only the simulated data provided by YOPO-Sim is used for training, while the planner can be used in real-world scenarios without fine-tuning.
• Figure 3: Visualization of the cone constraint, cost map, and expert trajectories. The cone constraint is defined by the primitive anchor norm $P_{n_{max}}$ and the direction vector $\bm{n}$. The offset $\Delta \bm{P}$ is constrained to the cone's interior. The cost map is generated by TTA, which evaluates the cost function in \ref{['eq:objective_function']}. The trajectory optimization problem is solved by CasADi CasADi. For a better illustration, please refer to the video.
• Figure 4: Snapshots of the YOPO-Sim. The insets show some of the sensors provided by the simulator: (a) depth camera and RGB camera, (b) LiDAR, and (c) point cloud map.
• Figure 5: The DIABLO vehicle platform for the real-world experiment is equipped with a RealSense T265 camera for odometry and an OAK-D-Pro stereo camera for depth image.