Learning Latent Action World Models In The Wild

TL;DR

The paper tackles predicting future outcomes in unconstrained real-world video by learning a latent action space without labeled actions. It introduces a latent action world model trained from large-scale in-the-wild videos using an inverse dynamics model to infer latent actions $z_t$ and a forward model to predict $s_{t+1}$, with continuous latents outperforming discrete quantization. Findings show that latent actions tend to be camera-relative and spatially localized, yet can transfer complex motions across videos; a lightweight controller mapping real actions to latents enables planning tasks with performance close to action-conditioned baselines. These results demonstrate the feasibility and scalability of latent-action world models for real-world reasoning and planning, suggesting practical pathways to broader domain transfer and multi-embodiment intelligence.

Abstract

Agents capable of reasoning and planning in the real world require the ability of predicting the consequences of their actions. While world models possess this capability, they most often require action labels, that can be complex to obtain at scale. This motivates the learning of latent action models, that can learn an action space from videos alone. Our work addresses the problem of learning latent actions world models on in-the-wild videos, expanding the scope of existing works that focus on simple robotics simulations, video games, or manipulation data. While this allows us to capture richer actions, it also introduces challenges stemming from the video diversity, such as environmental noise, or the lack of a common embodiment across videos. To address some of the challenges, we discuss properties that actions should follow as well as relevant architectural choices and evaluations. We find that continuous, but constrained, latent actions are able to capture the complexity of actions from in-the-wild videos, something that the common vector quantization does not. We for example find that changes in the environment coming from agents, such as humans entering the room, can be transferred across videos. This highlights the capability of learning actions that are specific to in-the-wild videos. In the absence of a common embodiment across videos, we are mainly able to learn latent actions that become localized in space, relative to the camera. Nonetheless, we are able to train a controller that maps known actions to latent ones, allowing us to use latent actions as a universal interface and solve planning tasks with our world model with similar performance as action-conditioned baselines. Our analyses and experiments provide a step towards scaling latent action models to the real world.

Figures (25)

• Figure 1: Action diversity. Classically used navigation or manipulation data contains the most general actions, such as camera or hand movements. In-the-wild videos extend this to a much broader distribution of actions, with objects entering the scene or people dancing.
• Figure 2: Latent action world model. A classical world model is endowed with actions represented as latent variables. These latent actions are obtained thanks to an inverse dynamics model trained jointly with the world model. To limit their information content (and propensity to cheat), they are regularized using techniques such as noise addition, sparsification, or quantization.
• Figure 3: Sample predictions using the IDM. We illustrate the highest quality unrollings obtained with different regularization, using the inverse dynamics model. While sparse or noisy latent actions are able to capture a man entering the scene, discrete ones are not able to properly capture such action, even if some motions remains captured.
• Figure 4: IDM performance. We report the one step prediction error on in-the-wild videos. Adjusting the capacity of sparsity and noise based latent actions allows for varying performance, while quantized ones struggle to adapt to the complexity.
• Figure 5: Raw latent evaluation. By artificially stitching videos, we can create abrupt scene changes. Measuring how the prediction error increases when such changes happen compared to the original video tells us how well the model can capture the whole next frame (a). To measure the transferability of latent actions, we measure if they inference is cycle-consistent. We infer latent actions on video A, then apply them of another random video. From this prediction, we re-infer the latent actions and apply them on video A. If the latent action transfers well, we should obtain a small error with video A (b). The combination of both metrics ensures that shortcuts are not the source of the transfer.