EVA: An Embodied World Model for Future Video Anticipation

TL;DR

This work tackles the limitation of video generation models in embodied settings by introducing Reflection-of-Generation (RoG), which interleaves understanding with generation to enable iterative self-correction and longer-horizon predictions. Building on RoG, Embodied Video Anticipator (EVA) fuses a multimodal visual language model with a diffusion-based video generator and uses chunk-wise autoregression, Ensemble-LoRA, and a cross-attention adapter to achieve robust, domain-adaptive video anticipation. The authors provide EVA-Bench, a comprehensive benchmark with in-domain and OOD data across four meta-tasks to evaluate both understanding and generation capabilities, including robot planning scenarios. Extensive experiments demonstrate EVA’s superior performance in video generation, VQA, and robotics tasks, highlighting RoG’s benefits for robust embodied world models and the potential for large-scale pre-trained models in real-world video prediction. Overall, the work lays a foundation for scalable, interpretable embodied prediction systems that can generalize across diverse tasks and domains.

Abstract

Video generation models have made significant progress in simulating future states, showcasing their potential as world simulators in embodied scenarios. However, existing models often lack robust understanding, limiting their ability to perform multi-step predictions or handle Out-of-Distribution (OOD) scenarios. To address this challenge, we propose the Reflection of Generation (RoG), a set of intermediate reasoning strategies designed to enhance video prediction. It leverages the complementary strengths of pre-trained vision-language and video generation models, enabling them to function as a world model in embodied scenarios. To support RoG, we introduce Embodied Video Anticipation Benchmark(EVA-Bench), a comprehensive benchmark that evaluates embodied world models across diverse tasks and scenarios, utilizing both in-domain and OOD datasets. Building on this foundation, we devise a world model, Embodied Video Anticipator (EVA), that follows a multistage training paradigm to generate high-fidelity video frames and apply an autoregressive strategy to enable adaptive generalization for longer video sequences. Extensive experiments demonstrate the efficacy of EVA in various downstream tasks like video generation and robotics, thereby paving the way for large-scale pre-trained models in real-world video prediction applications. The video demos are available at \hyperlink{https://sites.google.com/view/icml-eva}{https://sites.google.com/view/icml-eva}.

Figures (12)

• Figure 1: The illustration of the Reflection of Generation(RoG). Giving the instruction and observation as input, the world model gives a proper output with the combination of understanding, prediction, and generation.
• Figure 2: Meta-tasks of the embodied-video prediction. We present four meta-tasks, including Action-Description, Finish-Thinking, How-To, and Next-Step, for embodied video anticipation and build the related dataset, benchmark, and model.
• Figure 3: RoG inference pipeline of EVA with chunk-wise frame extension. Given the visual observation and human questions as input, EVA would first generate fixed frames of videos and related text answers. Then, the model prompts itself to check the task completion status; if the predicted video is not finished, EVA keeps generating the extended frames until the task completion judgment is true.
• Figure 4: Visualization results of the How-To, Next-Step, and Finsh-Thinking. Starting from a random statue, EVA can generate robot motion and human-ego motion according to the instructions. The first two continuous cases show the long-horizon generation ability of EVA; in the last example, EVA can generate video based on its reasoning results. We include more example results on the demo page and in the Appendix \ref{['sec:appendix_exp']}.
• Figure 5: A unified visual understanding and generation framework of EVA. The EVA introduces a visual projector in understanding LLM, an image conditional resampler in the generation model, trained a generation adapter as a text condition for denoising UNet, and added an Ensemble LoRA system for domain-specific generation. We train the EVA separately, including three stages of alignment and training.