MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstruction

TL;DR

The paper tackles the problem of learning action-centric latent actions from diverse human videos to improve vision-language-action pretraining, especially when ground-truth actions are unavailable. It introduces MVP-LAM, a latent-action model trained on time-synchronized multi-view videos with a cross-viewpoint reconstruction objective that reduces viewpoint-specific cues and promotes action-relevant dynamics. Empirically, MVP-LAM achieves higher mutual information with ground-truth actions and better action prediction on Bridge V2, and its latent actions lead to substantial gains in downstream manipulation on SIMPLER and LIBERO-Long after VLA pretraining. The work demonstrates that cross-viewpoint reconstruction can preserve transition information under viewpoint perturbations and that incorporating diverse human data further enhances action-centricity, pointing to practical benefits for scalable robot learning.

Abstract

Learning \emph{latent actions} from diverse human videos enables scaling robot learning beyond embodiment-specific robot datasets, and these latent actions have recently been used as pseudo-action labels for vision-language-action (VLA) model pretraining. To make VLA pretraining effective, latent actions should contain information about the underlying agent's actions despite the absence of ground-truth labels. We propose \textbf{M}ulti-\textbf{V}iew\textbf{P}oint \textbf{L}atent \textbf{A}ction \textbf{M}odel (\textbf{MVP-LAM}), which learns discrete latent actions that are highly informative about ground-truth actions from time-synchronized multi-view videos. MVP-LAM trains latent actions with a \emph{cross-viewpoint reconstruction} objective, so that a latent action inferred from one view must explain the future in another view, reducing reliance on viewpoint-specific cues. On Bridge V2, MVP-LAM produces more action-centric latent actions, achieving higher mutual information with ground-truth actions and improved action prediction, including under out-of-distribution evaluation. Finally, pretraining VLAs with MVP-LAM latent actions improves downstream manipulation performance on the SIMPLER and LIBERO-Long benchmarks.

Figures (9)

• Figure 1: Why viewpoint variation interferes with learning latent actions. Viewpoint variation acts as noise. Frame-to-frame visual differences reflect both interaction-driven state changes and viewpoint-dependent appearance changes (e.g., camera movements). Because these factors are entangled, the same underlying action can induce different visual transitions across viewpoints. This confounding makes it difficult to learn latent actions that are consistently predictive of the underlying control actions.
• Figure 2: MVP-LAM training with time-synchronized multi-view videos. (1) Self-viewpoint reconstruction (left): for each view $v$, frozen DINOv2 extracts features $(o_t^v,o_{t+1}^v)$. A spatiotemporal encoder produces a continuous latent $e_t^v$ that is vector-quantized into a discrete token $z_t^v$, and a decoder reconstructs $o_{t+1}^v$ from $(o_t^v,z_t^v)$. (2) Cross-viewpoint reconstruction (right): MVP-LAM swaps latent tokens across views (e.g., $z_t^{v_1}\leftrightarrow z_t^{v_2}$) while reconstructing each view’s future feature, encouraging $z_t$ to capture inherent transition information.
• Figure 3: Estimated mutual information.$\mathcal{I}(Z;A)$ on Bridge V2 with KSG, BA, and MINE estimators. For KSG, latent actions are randomly projected to $d{=}256$ prior to estimation. Higher is better. Error bars show standard deviation over four seeds.
• Figure 4: Linear probing result. NMSE of a linear layer predicting actions from latent actions. Bridge V2 is in-distribution; LIBERO (Spatial/Object/Goal/Long) is out-of-distribution. Lower is better. Error bars show standard deviation over four seeds.
• Figure 5: Overview of the VLM pretraining and VLA finetuning with example demonstrations. Left: sample observation sequences from SIMPLER and LIBERO-Long with natural language goal description. Right: (1) VLM Pretraining. Prismatic-7B VLM is pretrained to predict the discrete latent action token, which is produced by MVP-LAM, from an image and language instruction using a CE loss. (2) VLA Finetuning. VLA initializes from the pretrained VLM and finetunes on downstream demonstrations to predict robot actions.