End-to-end Listen, Look, Speak and Act

TL;DR

The paper introduces ELLSA, a fully end-to-end, full-duplex model that listens, looks, speaks and acts concurrently across vision, speech, text, and action. It proposes SA-MoE, a Self-Attention Mixture-of-Experts architecture that routes each modality to specialized experts and fuses them with a shared attention backbone. It demonstrates competitive performance on speech interaction and robot manipulation benchmarks while enabling novel capabilities such as speaking-while-acting, action barge-ins, and context-grounded VQA. By integrating perception and generation in real time, the work advances embodied interactive intelligence toward more natural, general autonomous systems.

Abstract

Human interaction is inherently multimodal and full-duplex: we listen while watching, speak while acting, and fluidly adapt to turn-taking and interruptions. Realizing these capabilities is essential for building models simulating humans. We present ELLSA (End-to-end Listen, Look, Speak and Act), which, to our knowledge, is the first full-duplex, end-to-end model that simultaneously perceives and generates across vision, text, speech, and action within a single architecture, enabling interaction patterns previously out of reach, yielding more natural, human-like behaviors. At its core is a novel SA-MoE architecture (Self-Attention Mixture-of-Experts) that routes each modality to specialized experts and fuses them through a unified attention backbone. This provides a generalizable solution for joint multimodal perception and concurrent generation, leveraging strong pre-trained components while enabling efficient modality integration and mitigating modality interference. On speech-interaction and robot-manipulation benchmarks, ELLSA matches modality-specific baselines, while uniquely supporting advanced multimodal and full-duplex behaviors such as dialogue and action turn-taking, defective instruction rejection, speaking-while-acting, context-grounded visual question answering, and action barge-ins. We contend that ELLSA represents a step toward more natural and general interactive intelligence, contributing to the broader pursuit of artificial general intelligence. All data, code and model checkpoints will be released upon acceptance.

Figures (8)

• Figure 1: (a) Overview of ELLSA. In ELLSA, different modalities are processed by different experts, and experts are integrated in SA-MoE architecture to enable modality interaction. (b) Streaming full-duplex MIMO interaction by an interleaved temporal multimodal sequence.
• Figure 2: Working mechanism of SA-MoE. Each modality is routed to its designated expert, and cross-modal interaction is achieved through the attention mechanism. During inference, all experts share a unified KV cache. By attending to the KV cache, each expert can integrate information across modalities and achieve coherent multimodal understanding.
• Figure 3: The training strategy of ELLSA. First train individual experts, then build SA-MoE backbone by integrating these experts, finally connect speech synthesizer. Across these stages, both training tasks and trainable parameters evolve to suit the model’s growing capabilities.
• Figure 4: An example of ELLSA’s advanced capabilities: starting from a spoken instruction, the model executes the action, engages in context-grounded VQA, and supports action barge-in. This instance demonstrates not only ELLSA’s core skills but also its unique advanced capabilities: its MIMO capacity to process multimodal inputs and outputs simultaneously, and its duplex capability to manage complex conversational dynamics such as turn-taking and interruptions.
• Figure 5: Defective instruction rejection (visual). ELLSA accurately identified the color of the plate in current scene.