LatBot: Distilling Universal Latent Actions for Vision-Language-Action Models

TL;DR

LatBot tackles the transferability bottleneck of vision-language-action models by learning universal latent actions that separate robot motion from environmental dynamics. It introduces a decoupled latent-action representation (scene vs. motion tokens) and a unified decoder that jointly reconstructs future frames and predicts inter-frame actions, with knowledge distillation to VLA models via action-alignment and reasoning-preservation losses. The approach yields state-of-the-art results across SIMPLER, LIBERO, and real-world Franka tasks, including strong few-shot performance, and is supported by extensive ablations and analysis of latent-action contributions. This work demonstrates that incorporating physical priors and disentangling motion from scene changes significantly enhances cross-domain generalization for robotic manipulation.

Abstract

Learning transferable latent actions from large-scale object manipulation videos can significantly enhance generalization in downstream robotics tasks, as such representations are agnostic to different robot embodiments. Existing approaches primarily rely on visual reconstruction objectives while neglecting physical priors, leading to sub-optimal performance in learning universal representations. To address these challenges, we propose a Universal Latent Action Learning framework that takes task instructions and multiple frames as inputs, and optimizes both future frame reconstruction and action sequence prediction. Unlike prior works, incorporating action predictions (e.g., gripper or hand trajectories and orientations) allows the model to capture richer physical priors such as real-world distances and orientations, thereby enabling seamless transferability to downstream tasks. We further decompose the latent actions into learnable motion and scene tokens to distinguish the robot's active movements from environmental changes, thus filtering out irrelevant dynamics. By distilling the learned latent actions into the latest VLA models, we achieve strong performance across both simulated (SIMPLER and LIBERO) and real-world robot settings. Notably, with only 10 real-world trajectories per task collected on a Franka robot, our approach successfully completes all five challenging tasks, demonstrating strong few-shot transferability in robotic manipulation.

Figures (4)

• Figure 1: Different paradigms in latent action modeling (LAM). Existing methods often ignore disentangling robot actions from environmental changes. In contrast, we learn a disentangled representation and decode latent actions into both the future visual frame $V_{t+k}$ and physical actions $A_{t:t+k}$ that enables more accurate and transferable control for downstream tasks.
• Figure 2: Illustration of the proposed latent action distillation approach for VLA models. By optimizing the VLMs with latent action alignment loss and reasoning preservation loss, we distill generalizable action representations learned from both robot and human hand demonstration videos, while simultaneously maintaining sub-task planning capabilities. This is followed by an action expert module for continuous action prediction.
• Figure 3: The real-robot Franka setup is equipped with multi-view observations. Tasks include pick-up, insertion, and so on, requiring both translational and rotational motions. In addition, we evaluate more practical scenarios involving interactions with real objects such as thin brushes, heavy frying pans, and real ovens.
• Figure 4: Analysis of the effects the latent-action distillation on the VLM. Following kang2025your, we visualize the attention maps between the final text token and the visual features for both $\pi_{0.5}$ and our model across different real-robot tasks. The red bounding boxes in the RGB images mark the task-specific target, while the red boxes on our model’s attention maps highlight the regions with the strongest activations. The results show that after latent-action distillation, the VLM of our model exhibits enhanced spatial grounding capabilities, with its attention maps consistently concentrated within the red box.