VITA-E: Natural Embodied Interaction with Concurrent Seeing, Hearing, Speaking, and Acting

TL;DR

VITA-E tackles rigid, sequential interaction in vision-language-action systems by introducing a dual-model architecture and a model-as-controller paradigm that enables concurrent seeing, hearing, speaking, and acting with instant interruptibility. A VLM, guided by a learned control language emitting tokens such as $[RES]$, $[ACT]$, and $[HALT]$, directs a diffusion-based action expert to perform low-level motor control, while a Standby model can preemptively intervene during ongoing tasks. Experiments on a Fourier GR2 humanoid demonstrate strong real-time interaction capabilities, including perfect speech interruption and emergency stops, along with substantial concurrent speech and action performance, albeit with a noted gap to end-to-end pretraining baselines. Overall, the framework advances embodied AI by enabling more natural, responsive assistants capable of real-time collaboration with humans, at the cost of higher computational demands and the need for broader universality validation.

Abstract

Current Vision-Language-Action (VLA) models are often constrained by a rigid, static interaction paradigm, which lacks the ability to see, hear, speak, and act concurrently as well as handle real-time user interruptions dynamically. This hinders seamless embodied collaboration, resulting in an inflexible and unresponsive user experience. To address these limitations, we introduce VITA-E, a novel embodied interaction framework designed for both behavioral concurrency and nearly real-time interruption. The core of our approach is a dual-model architecture where two parallel VLA instances operate as an ``Active Model'' and a ``Standby Model'', allowing the embodied agent to observe its environment, listen to user speech, provide verbal responses, and execute actions, all concurrently and interruptibly, mimicking human-like multitasking capabilities. We further propose a ``model-as-controller'' paradigm, where we fine-tune the VLM to generate special tokens that serve as direct system-level commands, coupling the model's reasoning with the system's behavior. Experiments conducted on a physical humanoid platform demonstrate that VITA-E can reliably handle complex interactive scenarios. Our framework is compatible with various dual-system VLA models, achieving an extremely high success rate on emergency stops and speech interruptions while also successfully performing concurrent speech and action. This represents a significant step towards more natural and capable embodied assistants.

Figures (6)

• Figure 1: VITA-E is capable of handling a variety of complex interactive scenarios, including nearly real-time concurrency and interruption. Please see our demo video at this https://youtu.be/jplQ0R50kfU.
• Figure 2: VITA-E's responses and actions in various interactive scenarios and instructions.
• Figure 3: The logical architecture and operational states of the active model in our dual-model VITA-E framework. Each of the two models can switch between "Active" and "Standby" states. When a model becomes active, the VLM part acts as a controller, processing user inputs in the "Hearing" state and generating special tokens that can trigger a transition to the "Action" state, where it collaborates with an action expert as a whole VLA model. For example, when the VLM's output starts with the [ACT] token, the text between the [ACT] token and the [INST] token will be played as audio, and the text after the [INST] token will be sent to the VLA model as an action instruction. Otherwise, the text after the special token will only be played as audio, and the system will execute the command corresponding to the special token. For the detailed functions of each special token, see Table \ref{['tab:special-token-details']}.
• Figure 4: Sequence diagrams illustrating the four primary interaction modes. Model I is the Active Model, and Model II is the Standby Model. V and A represent voice and action generation, respectively. The Standby Model can process new requests in parallel or preempt the Active Model to handle interruptions and task switches.
• Figure 5: Success rate comparison of VITA-E and GR00T on the Libero benchmark. Although the success rates of VITA-E on this benchmark are not as good as the baseline with the same structure, we emphasize that the goal of this work is not for VITA-E to achieve the state-of-the-art experimental performance, but to prove that the model is capable of completing embodied tasks and to provide quantitative metrics.