EgoActor: Grounding Task Planning into Spatial-aware Egocentric Actions for Humanoid Robots via Visual-Language Models

TL;DR

EgoActor presents a unified vision-language approach to grounding high-level instructions into egocentric, executable actions for humanoid robots, addressing the gap between abstract planning and real-world motor control. By introducing EgoActing as the task and leveraging a scalable VLM (EgoActor) that jointly predicts locomotion, head movements, manipulation, and human interaction, the method achieves sub-second inference on both 4B and 8B models. The training data blend real-world egocentric videos, spatial reasoning, and simulated trajectories, enabling robust generalization to unseen layouts and objects across real and virtual benchmarks. The work demonstrates improved traversal, manipulation, and interaction capabilities in complex environments and offers open-source resources to foster reproducibility and further research.

Abstract

Deploying humanoid robots in real-world settings is fundamentally challenging, as it demands tight integration of perception, locomotion, and manipulation under partial-information observations and dynamically changing environments. As well as transitioning robustly between sub-tasks of different types. Towards addressing these challenges, we propose a novel task - EgoActing, which requires directly grounding high-level instructions into various, precise, spatially aware humanoid actions. We further instantiate this task by introducing EgoActor, a unified and scalable vision-language model (VLM) that can predict locomotion primitives (e.g., walk, turn, move sideways, change height), head movements, manipulation commands, and human-robot interactions to coordinate perception and execution in real-time. We leverage broad supervision over egocentric RGB-only data from real-world demonstrations, spatial reasoning question-answering, and simulated environment demonstrations, enabling EgoActor to make robust, context-aware decisions and perform fluent action inference (under 1s) with both 8B and 4B parameter models. Extensive evaluations in both simulated and real-world environments demonstrate that EgoActor effectively bridges abstract task planning and concrete motor execution, while generalizing across diverse tasks and unseen environments.

Figures (9)

• Figure 1: Overview of EgoActor, which can control a humanoid robot by jointly predicting movement, active perception, manipulation, and human interaction actions to achieve coordinated and precise execution, enabling humanoid robots to conduct long-horizon multi-step task instructions described in natural language.
• Figure 2: Visualization of EgoActor's working procedure for a given task: "Approach and pick up the orange on the desk". The grey blocks represent structured language actions (SLAs) and the green blocks represent natural language actions (NLAs).
• Figure 3: Example natural language actions (NLA) in EgoActing. EgoActor is trained to predict the corresponding actions based on obtained RGB observations.
• Figure 4: Multi-step illustration of obstacle avoidance generalization of our model, when faced with an unseen string obstacle.
• Figure 5: First-person view of an EgoActor's active perception trace. Color description blocks highlight model's behaviors.