LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instruction

TL;DR

LeVERB introduces a latent vision-language interface for humanoid whole-body control by decoupling high-level vision-language reasoning (System 2) from low-level dynamics (System 1) through a learned latent verb vocabulary. It pairs a CVAE-based LeVERB-VL high-level policy with a distillation-based LeVERB-A low-level controller, trained entirely on synthetic data but capable of zero-shot deployment on real hardware. A photorealistic LeVERB-Bench provides 150+ tasks across 10 categories to evaluate closed-loop performance and sim-to-real transfer, achieving 58.9% average success and up to 80% on simpler visual navigation tasks, outperforming naive baselines by 7.8×. The work advances vision-language grounded robotic control by enabling expressive, scene-aware whole-body motions and demonstrating first zero-shot sim-to-real results for humanoid WBC with a latent action interface.

Abstract

Vision-language-action (VLA) models have demonstrated strong semantic understanding and zero-shot generalization, yet most existing systems assume an accurate low-level controller with hand-crafted action "vocabulary" such as end-effector pose or root velocity. This assumption confines prior work to quasi-static tasks and precludes the agile, whole-body behaviors required by humanoid whole-body control (WBC) tasks. To capture this gap in the literature, we start by introducing the first sim-to-real-ready, vision-language, closed-loop benchmark for humanoid WBC, comprising over 150 tasks from 10 categories. We then propose LeVERB: Latent Vision-Language-Encoded Robot Behavior, a hierarchical latent instruction-following framework for humanoid vision-language WBC, the first of its kind. At the top level, a vision-language policy learns a latent action vocabulary from synthetically rendered kinematic demonstrations; at the low level, a reinforcement-learned WBC policy consumes these latent verbs to generate dynamics-level commands. In our benchmark, LeVERB can zero-shot attain a 80% success rate on simple visual navigation tasks, and 58.5% success rate overall, outperforming naive hierarchical whole-body VLA implementation by 7.8 times.

Figures (5)

• Figure 1: Overview of our contributions. Top: we create a photorealistic and dynamically accurate benchmark for humanoid vision-language WBC. Middle: in real world, we zero-shot deploy a dual-process VLA model trained only on synthetic data. Bottom: a high-level overview of our model architecture with decoupled vision-language and dynamics-level action processing.
• Figure 2: Visualization of LeVERB-Bench environments. Top row: hundreds of texture and object randomization options. Middle row: egocentric camera view and randomized third-person camera views. Bottom row: diverse task categories.
• Figure 3: Details of our data collection and training pipeline. Step 1: we collect a synthetic, photorealistic dataset of retargeted motions in IsaacSim, and annotate with text instructions. Step 2: we train LeVERB-VL with a kinematic trajectory reconstruction task, and obtain a regularized latent verb vocabulary, from which we cache the latent verbs $z_t^{(i)}$ for every rollout $i$ in the dataset. Step 3: we use $z_t^{(i)}$ to condition LeVERB-A. It is DAgger-distilled from teacher tracking policy $T_\xi$, which receives future reference command $s_{t+1}$ that corresponds to the latent verb's intention.
• Figure 4: Top: LeVERB responds robustly to human vocabulary variations. Bottom: LeVERB executes different sit-down maneuvers conditioned on the chair's visual location, demonstrating spatial reasoning capabilities.
• Figure 5: The validation loss curves for training of system 2 of LeVERB The left part shows the validation loss curve in Equation \ref{['eq:full_loss']} with different size of backbone Transformer. The right part shows the validation loss of trajectory reconstruction for conditioned on different input.