LaST$_{0}$: Latent Spatio-Temporal Chain-of-Thought for Robotic Vision-Language-Action Model

TL;DR

LaST$_{0}$ tackles the latency and representational bottlenecks of explicit CoT in Vision–Language–Action for robotic manipulation by introducing a Latent Spatio-Temporal CoT (LaST CoT) and a dual Mixture-of-Transformers framework that separates slow latent reasoning from fast action generation. It constructs a token-efficient latent space that jointly encodes future visual dynamics, 3D geometry, and proprioception across a horizon $H$, enabling temporally coherent internal reasoning trajectories. The slow reasoning expert autoregressively predicts latent CoT tokens, supervised by cosine similarity to ground-truth latent targets, while the fast acting expert uses Flow Matching to generate high-frequency actions conditioned on updated latent representations, with asynchronous updates controlled by $\kappa$ and KV-cache acceleration achieving real-time performance (e.g., $15.4$ Hz). Through large-scale pretraining on $\sim$400k trajectories and staged supervised fine-tuning, LaST$_{0}$ yields state-of-the-art results on RLBench (mean SR $=82\%$) and six real-world tasks, with substantial speedups over explicit CoT methods and robust long-horizon manipulation capabilities. This approach demonstrates scalable, physically grounded reasoning for robotic VLA, offering practical impact for real-time, complex manipulation in dynamic environments.

Abstract

Vision-Language-Action (VLA) models have recently demonstrated strong generalization capabilities in robotic manipulation. Some existing VLA approaches attempt to improve action accuracy by explicitly generating linguistic reasoning traces or future visual observations before action execution. However, explicit reasoning typically incurs non-negligible inference latency, which constrains the temporal resolution required for robotic manipulation. Moreover, such reasoning is confined to the linguistic space, imposing a representational bottleneck that struggles to faithfully capture ineffable physical attributes. To mitigate these limitations, we propose LaST$_0$, a framework that enables efficient reasoning before acting through a Latent Spatio-Temporal Chain-of-Thought (CoT), capturing fine-grained physical and robotic dynamics that are often difficult to verbalize. Specifically, we introduce a token-efficient latent CoT space that models future visual dynamics, 3D structural information, and robot proprioceptive states, and further extends these representations across time to enable temporally consistent implicit reasoning trajectories. Furthermore, LaST$_0$ adopts a dual-system architecture implemented via a Mixture-of-Transformers design, where a reasoning expert conducts low-frequency latent inference and an acting expert generates high-frequency actions conditioned on robotics-oriented latent representations. To facilitate coordination, LaST$_0$ is trained with heterogeneous operation frequencies, enabling adaptive switching between reasoning and action inference rates during deployment. Across ten simulated and six real-world manipulation tasks, LaST$_0$ improves mean success rates by 8% and 13% over prior VLA methods, respectively, while achieving substantially faster inference. Project website: https://sites.google.com/view/last0

Figures (8)

• Figure 1: Overview. (a) Unlike previous VLA methods that seek to improve manipulation by explicitly generating linguistic reasoning traces or future visual observations, (b) we propose LaST$_0$, a framework that enables efficient reasoning before acting through a Latent Spatio-Temporal CoT. This latent CoT captures multimodal physical and robotic dynamics that are difficult to verbalize and propagates them over time to form temporally consistent reasoning trajectories. LaST$_0$ achieves SOTA performance across a wide range of real-world and simulated tasks, while supporting real-time model inference.
• Figure 2: Overall framework of LaST$_{0}$.a) We propose LaST$_{0}$, a unified VLA model with a dual-system architecture. The model is implemented via a MoT scheme with two experts interacting through shared self-attention. The slow reasoning expert operates at a low frequency, taking visual observations and text as input to construct the LaST CoT by autoregressively predicting spatio-temporal latent tokens, which are stored in the KV cache. The fast acting expert operates at a higher frequency and generates actions via flow matching, conditioned on high-frequency observations and periodically updated latent representations. b) We design a spatio-temporal latent space tailored for the reasoning expert, where pretrained modality-specific encoders extract features from future RGB images, point clouds, and robot states at keyframes. These features serve as ground-truth latent CoT targets for supervising the reasoning expert. c) The training procedure consists of three stages with different parameter update strategies, ensuring a reliable latent and robust action generation.
• Figure 3: The slow reasoning expert performs low-frequency latent CoT reasoning to capture long-horizon spatio-temporal dependencies, while the fast acting expert generates actions conditioned on high-frequency observations and periodically updated latent knowledge. Benefiting from training with a mixture of fast–slow operating ratios, the model can flexibly adjust the reasoning–action frequency ratio at test time.
• Figure 4: Ablation study on key design choices of LaST$_0$. We analyze (a) the importance of different latent modalities, (b) the number of tokens allocated per latent modality, (c) the temporal coverage in latent reasoning, and (d) the collaboration frequency between reasoning and action experts. Results are reported as average success rates across 10 RLBench tasks, demonstrating the contribution of each component to overall performance.
• Figure 5: Attention heatmap visualizations from the last layer for three VLA models: (a) LaST$_0$ without CoT reasoning, (b) the explicit CoT in CoT-VLA, and (c) LaST$_0$ with LaST CoT.