TTT-Parkour: Rapid Test-Time Training for Perceptive Robot Parkour

TL;DR

TTT-Parkour presents a real-to-sim-to-real framework enabling rapid adaptation of humanoid robots to unseen, highly challenging terrains through rapid test-time training. It combines a two-stage learning process—pre-training on procedurally generated terrains and fast fine-tuning on high-fidelity RGB-D reconstructed meshes—with a fast geometry reconstruction pipeline that provides simulation-ready meshes via scale recovery and frame alignment. The approach achieves robust zero-shot sim-to-real transfer across wedges, stakes, boxes, trapezoids, and narrow beams, with adaptation completing in about 10 minutes and a significant gap over training-from-scratch baselines. This work narrows the sim-to-real gap for dynamic humanoid parkour and offers a practical path to rapid deployment in complex environments.

Abstract

Achieving highly dynamic humanoid parkour on unseen, complex terrains remains a challenge in robotics. Although general locomotion policies demonstrate capabilities across broad terrain distributions, they often struggle with arbitrary and highly challenging environments. To overcome this limitation, we propose a real-to-sim-to-real framework that leverages rapid test-time training (TTT) on novel terrains, significantly enhancing the robot's capability to traverse extremely difficult geometries. We adopt a two-stage end-to-end learning paradigm: a policy is first pre-trained on diverse procedurally generated terrains, followed by rapid fine-tuning on high-fidelity meshes reconstructed from real-world captures. Specifically, we develop a feed-forward, efficient, and high-fidelity geometry reconstruction pipeline using RGB-D inputs, ensuring both speed and quality during test-time training. We demonstrate that TTT-Parkour empowers humanoid robots to master complex obstacles, including wedges, stakes, boxes, trapezoids, and narrow beams. The whole pipeline of capturing, reconstructing, and test-time training requires less than 10 minutes on most tested terrains. Extensive experiments show that the policy after test-time training exhibits robust zero-shot sim-to-real transfer capability.

Figures (14)

• Figure 1: Rapid test-time training on unseen terrain. By reconstructing the scene and fine-tuning in simulation, our framework enables the robot to master challenging obstacles within 10 minutes, turning failure (left) into success (right).
• Figure 2: TTT-Parkour. Our framework consists of three stages: (1) Pre-training: A general policy is pre-trained on diverse procedurally generated terrains to learn robust locomotion primitives. (2) Test-time Training (TTT): We reconstruct high-fidelity and simulation-ready meshes from real-world captures using feed-forward reconstruction with automatic scale recovery and frame alignment. The policy is then rapidly fine-tuned on these specific terrains in simulation. (3) Sim-to-Real Deployment: The adapted policy is directly deployed to the real-world humanoid robot for zero-shot traversal of complex unseen obstacles.
• Figure 3: Efficient Geometry reconstruction. Our pipeline consists of four stages: (1) Real-World Capture. (2) Feed-forward Reconstruction provides initial scene geometry from RGB sequences. (3) Scale Recovery corrects metric scale discrepancies by aligning inferred depth with sensor depth. (4) Physically-consistent Frame Alignment registers the terrain to the simulation coordinate system by aligning the $z$-axis with gravity and the $x$-axis with the traversal direction using 3D semantic segmentation.
• Figure 4: Real-world experiments. The robot successfully traverses extremely challenging terrains, including: (a) Wedges, (b) Stakes, (c) Boxes, (d) Trapezoids, (e) Narrow beam, and (f) Mixed terrain. See https://ttt-parkour.github.io for more.
• Figure 5: Success rate progression over test-time training (TTT-1) iterations. The policy rapidly converges to high performance on previously unseen terrains.