Olaf-World: Orienting Latent Actions for Video World Modeling

TL;DR

This work addresses the lack of transferable latent actions when learning from unlabeled video by identifying cross-context non-identifiability in traditional stepwise latent action models. It introduces Seq$\Delta$-REPA, a sequence-level control-to-effect alignment that anchors integrated latent actions to observable semantic changes derived from a frozen self-supervised video encoder, yielding context-invariant action semantics. Building on this, Olaf-World pretrains action-conditioned video world models from passive video, enabling zero-shot action transfer and data-efficient adaptation to new control interfaces via a lightweight action adapter and LoRA. The approach demonstrates improved latent-space structure, stronger cross-context transfer, and robust generalization to unseen contexts, offering scalable, label-free pretraining for controllable video modeling with practical impacts for robotics and animation.

Abstract

Scaling action-controllable world models is limited by the scarcity of action labels. While latent action learning promises to extract control interfaces from unlabeled video, learned latents often fail to transfer across contexts: they entangle scene-specific cues and lack a shared coordinate system. This occurs because standard objectives operate only within each clip, providing no mechanism to align action semantics across contexts. Our key insight is that although actions are unobserved, their semantic effects are observable and can serve as a shared reference. We introduce Seq$Δ$-REPA, a sequence-level control-effect alignment objective that anchors integrated latent action to temporal feature differences from a frozen, self-supervised video encoder. Building on this, we present Olaf-World, a pipeline that pretrains action-conditioned video world models from large-scale passive video. Extensive experiments demonstrate that our method learns a more structured latent action space, leading to stronger zero-shot action transfer and more data-efficient adaptation to new control interfaces than state-of-the-art baselines.

Figures (11)

• Figure 1: We present Olaf-World, an adaptable video world model pretrained with transferable latent actions learned via Seq$\Delta$-REPA, enabling (A) context-invariant zero-shot action transfer, (B) efficient adaptation to new action spaces with minimal labeled data (e.g., 1 minute), and (C) improved generalization to novel scenes. Readers can click and play the video clips in this figure using Adobe Acrobat.
• Figure 2: Latent action learning.Problem: transition-based latent action models (LAMs) can reconstruct well, but fail to transfer (the same semantic action, e.g., "Forward", maps to different latent directions across contexts). Cause: the latent space is identified only up to a clip-specific basis, so there is no shared coordinate system. Solution: Seq$\Delta$-REPA uses the observable effect direction $\Delta y$ from a frozen video encoder as a shared reference and aligns latent actions to it, yielding consistent action semantics across contexts.
• Figure 3: Overall pipeline. (a) We train a latent action model (LAM) and encourage cross-context consistency by aligning action effects in a frozen video-feature space using Seq$\Delta$-REPA. (b) We then apply the frozen LAM to unlabeled videos to extract latent-action sequences, and use them as a unified control interface to pretrain an action-conditioned video world model.
• Figure 4: In-/cross-domain linear probing over training. Solid: in-domain; dashed: cross-domain evaluation (1st-P$\leftrightarrow$3rd-P). Seq$\Delta$-REPA consistently improves both in-domain and cross-domain probe performance.
• Figure 5: Cross-domain action similarity. Cosine similarity between per-action prototypes from 1st-P (rows) and 3rd-P (columns). Seq$\Delta$-REPA produces a more diagonal-dominant matrix (stronger one-to-one matching across contexts).

Theorems & Definitions (2)

• Proposition 1.1
• proof