Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding

TL;DR

This work tackles dynamic 3D scene understanding from egocentric observations by introducing Embodied VideoAgent, a memory-augmented multimodal agent that fuses egocentric video with embodied sensory inputs. It builds a persistent object memory $O$ and uses a VLM-based memory updater to keep scene state current as actions unfold, while an LLM-based framework orchestrates tool use and embodied actions. The approach is evaluated on Ego4D-VQ3D, OpenEQA, and EnvQA, showing consistent gains over end-to-end multimodal models and other agents, with improvements of $4.9\%$, $5.8\%$, and $11.7\%$ on the respective benchmarks. The results demonstrate the method’s ability to support precise object localization, embodied question answering, and robot manipulation tasks, highlighting its potential for robust, memory-driven embodied AI in dynamic environments.

Abstract

This paper investigates the problem of understanding dynamic 3D scenes from egocentric observations, a key challenge in robotics and embodied AI. Unlike prior studies that explored this as long-form video understanding and utilized egocentric video only, we instead propose an LLM-based agent, Embodied VideoAgent, which constructs scene memory from both egocentric video and embodied sensory inputs (e.g. depth and pose sensing). We further introduce a VLM-based approach to automatically update the memory when actions or activities over objects are perceived. Embodied VideoAgent attains significant advantages over counterparts in challenging reasoning and planning tasks in 3D scenes, achieving gains of 4.9% on Ego4D-VQ3D, 5.8% on OpenEQA, and 11.7% on EnvQA. We have also demonstrated its potential in various embodied AI tasks including generating embodied interactions and perception for robot manipulation. The code and demo will be made public.

Figures (15)

• Figure 1: Embodied VideoAgent is a multimodal agent that 1) builds scene memory from both egocentric video and embodied sensory input; 2) utilizes multiple tools to query this memory; 3) activates embodied action primitives to interact with the environments, effectively fulfills various user requests.
• Figure 2: An overview of Embodied VideoAgent. Left: We first translate the egocentric video and embodied sensory input (depth maps and camera poses) into structured representations: persistent object memory and history buffer. While the memory can be updated using VLM to support dynamic scenes where actions are being performed constantly; Right: the LLM within Embodied VideoAgent is prompted to fulfill the user's request by interactively invoking tools to query the memory and calling embodied action primitives to complete the task.
• Figure 3: Visualization of the entries in persistent object memory$\mathcal{M}_O$. Each 3D bounding box corresponds to an entry in the memory. As the video proceeds, objects (e.g. the large canned tomato paste) can be tracked/re-IDed and have their memory entries updated.
• Figure 4: An illustration of our VLM-based memory update method. This approach effectively prompts the VLM to associate an action with relevant object entries in memory through visual prompting, identifying the entries corresponding to the action’s target objects.
• Figure 5: An overview of our synthetic embodied data collection framework. An LLM plays the user role and is prompted to propose engaging tasks based on a partial scene graph and the user's feedback, while the user, effectively a Embodied VideoAgent, explores the scene and fulfills the user's requests.