Vision-Language Models Unlock Task-Centric Latent Actions

TL;DR

This paper addresses latent-action learning under action-correlated distractors in vision-language-action models by introducing promptable representations from Vision-Language Models as unsupervised targets for latent-action training. The approach removes distractor-induced noise, enabling LAPO to regain ground-truth actions and yields substantial downstream gains on Distracting MetaWorld MT10, up to sixfold improvements when distractors are present. A large-scale VLM benchmark reveals that representation quality depends strongly on language conditioning and prompt design rather than model size or architecture, with Molmo often providing the best, most robust signals. The work demonstrates a scalable path to task-centric latent actions and invites broader evaluation of promptable representations across robotics benchmarks and VLA architectures.

Abstract

Latent Action Models (LAMs) have rapidly gained traction as an important component in the pre-training pipelines of leading Vision-Language-Action models. However, they fail when observations contain action-correlated distractors, often encoding noise instead of meaningful latent actions. Humans, on the other hand, can effortlessly distinguish task-relevant motions from irrelevant details in any video given only a brief task description. In this work, we propose to utilize the common-sense reasoning abilities of Vision-Language Models (VLMs) to provide promptable representations, effectively separating controllable changes from the noise in unsupervised way. We use these representations as targets during LAM training and benchmark a wide variety of popular VLMs, revealing substantial variation in the quality of promptable representations as well as their robustness to different prompts and hyperparameters. Interestingly, we find that more recent VLMs may perform worse than older ones. Finally, we show that simply asking VLMs to ignore distractors can substantially improve latent action quality, yielding up to a six-fold increase in downstream success rates on Distracting MetaWorld.

Figures (16)

• Figure 1: Main result. Success rate on MetaWorld-10 benchmark for LAPO and proposed LAPO+VLM (Molmo), which uses promptable representations. We use three random seeds and report IQM and $95\%$-CI based on stratified bootstrapping, following the agarwal2021deep. See \ref{['sec:sc-rate']} for full results.
• Figure 2: Visualization of the task-relevant promptable representations extraction from the VLMs and their subsequent use as targets during latent action learning.
• Figure 3: Visualization of observations with and without distractors in our modification of MetaWorld environment.
• Figure 4: Demonstration that quality of latent actions learned by LAPO completely degrades in the presence of distractors, which results in almost zero success rate. We show that with the ideal target for FDM, which perfectly disentangles controllable features from the noise, performance may be restored, serving as a main motivation for us to explore promptable representations. Action probe represents MSE of a linear probe trained to predict real actions from latent actions. See \ref{['sec:setup']} for detailed explanation. We use three random seeds and report IQM and $95\%$-CI based on stratified bootstrapping, following the agarwal2021deep. See \ref{['sec:lapo-twin']} for details.
• Figure 5: Baseline LAPO action probes on MT10. Averaged over 3 random seeds. Action probe represents MSE of a linear probe trained to predict real actions from latent actions. See \ref{['sec:setup']} for detailed explanation.