Active Intelligence in Video Avatars via Closed-loop World Modeling

TL;DR

This work targets the gap between passive video-avatar generation and autonomous, goal-directed behavior by introducing L-IVA, a benchmark for long-horizon task completion in interactive environments, and ORCA, an IWM-inspired framework with a closed-loop OTAR cycle and a hierarchical dual-system architecture. By formulating avatar control as a POMDP and continuously updating beliefs through outcome-verified generations, ORCA achieves robust, multi-step task execution in stochastic generative worlds. The method demonstrates superior task success and behavioral coherence compared with open-loop and non-reflective baselines, validating the IWM-inspired design for active intelligence in video avatars. The work lays a foundation for deploying autonomous virtual hosts, livestreamers, and interactive agents where long-term planning and adaptive environment interaction are essential.

Abstract

Current video avatar generation methods excel at identity preservation and motion alignment but lack genuine agency, they cannot autonomously pursue long-term goals through adaptive environmental interaction. We address this by introducing L-IVA (Long-horizon Interactive Visual Avatar), a task and benchmark for evaluating goal-directed planning in stochastic generative environments, and ORCA (Online Reasoning and Cognitive Architecture), the first framework enabling active intelligence in video avatars. ORCA embodies Internal World Model (IWM) capabilities through two key innovations: (1) a closed-loop OTAR cycle (Observe-Think-Act-Reflect) that maintains robust state tracking under generative uncertainty by continuously verifying predicted outcomes against actual generations, and (2) a hierarchical dual-system architecture where System 2 performs strategic reasoning with state prediction while System 1 translates abstract plans into precise, model-specific action captions. By formulating avatar control as a POMDP and implementing continuous belief updating with outcome verification, ORCA enables autonomous multi-step task completion in open-domain scenarios. Extensive experiments demonstrate that ORCA significantly outperforms open-loop and non-reflective baselines in task success rate and behavioral coherence, validating our IWM-inspired design for advancing video avatar intelligence from passive animation to active, goal-oriented behavior.

Figures (9)

• Figure 1: Comparison of video avatar generation approaches. (a) Speech-driven and pose-driven methods produce passive motions with limited semantic understanding. In contrast, (b) our Online Reasoning and Cognitive Architecture (ORCA) enables complex, multi-step task execution through OTAR (Observe-Think-Act-Reflect) closed-loop reasoning.
• Figure 2: Overview of the ORCA framework. ORCA operates through a closed-loop OTAR cycle: Observe updates internal world state from generated clips, Think (System 2) decomposes tasks and predict next state, Act (System 1) translates subgoals into action captions for I2V generation, and Reflect verifies completion to accept/reject outcomes. This hierarchical dual-system architecture enables robust long-horizon task execution through continuous state tracking and adaptive replanning.
• Figure 3: L-IVA Benchmark Overview.Top: Statistical analysis showing (left) balanced scene distribution across 5 categories, (center) data source composition with 92 synthetic and 8 real images, and (right) task complexity distribution averaging 5.0 sub-goals per task. Bottom: Representative scenes from our benchmark including Garden, Kitchen, and livestream scenarios, demonstrating diverse real-world settings requiring multi-step object interactions.
• Figure 4: Qualitative Comparison on Transfer Plant Task. We compare four methods on long-horizon video generation. Top: Ground truth subgoals for reference. Red boxes indicate execution failures or error accumulation. Open-Loop planner cannot detect execution errors. Reactive agent lacks world state knowledge, leading to repetitive actions. VAGEN's I2V errors corrupt the final state without reflection. ORCA (Ours) successfully completes all subgoals with consistent execution quality.
• Figure 5: Overview of the L-IVA Benchmark Construction Pipeline. (a) Our data curation process employs a hybrid strategy: Pipeline A sources real-world images from Pexels, filtered by scene affordance and annotated via Gemini-2.5-Pro. Pipeline B utilizes a goal-first design for synthetic data, where scenes are generated by Nanobanana to strictly align with intended interactions. (b) A representative scene image (e.g., "Checking beehives") from the benchmark. (c) The corresponding structured metadata (YAML), including object inventory, high-level intention, subgoals, and reference prompts.