VLA-4D: Embedding 4D Awareness into Vision-Language-Action Models for SpatioTemporally Coherent Robotic Manipulation

TL;DR

VLA-4D addresses the challenge of spatiotemporal coherence in vision-language-action robotic manipulation by introducing 4D-aware visual representations and spatiotemporal action representations. It embeds 3D positions and 1D time into visual features using a spatiotemporal embedding and cross-attention, and augments action parameters with temporal variables aligned to a language model head. The method is trained in two stages with an extended LIBERO dataset that includes temporal action annotations, achieving state-of-the-art performance and smoother, more coherent manipulation trajectories. This work advances embodied AI by integrating space-time reasoning and multimodal alignment to enable robust, temporally consistent robotic control.

Abstract

Vision-language-action (VLA) models show potential for general robotic tasks, but remain challenging in spatiotemporally coherent manipulation, which requires fine-grained representations. Typically, existing methods embed 3D positions into visual representations to enhance the spatial precision of actions. However, these methods struggle to achieve temporally coherent control over action execution. In this work, we propose VLA-4D, a general VLA model with 4D awareness for spatiotemporally coherent robotic manipulation. Our model is guided by two key designs: 1) 4D-aware visual representation. We extract visual features, embed 1D time into 3D positions for 4D embeddings, and fuse them into a unified visual representation via a cross-attention mechanism. 2) Spatiotemporal action representation. We extend conventional spatial action representations with temporal information to enable the spatiotemporal planning, and align the multimodal representations into the LLM for spatiotemporal action prediction. Within this unified framework, the designed visual and action representations jointly make robotic manipulation spatially-smooth and temporally-coherent. In addition, we extend the VLA dataset with temporal action annotations for fine-tuning our model. Extensive experiments have been conducted to verify the superiority of our method across different tasks of robotic manipulation.

Figures (7)

• Figure 1: Illustration of various VLA paradigms for robotic manipulation. (a) 2D VLAs encode visual and language modalities into the LLM to predict actions, which remain spatiotemporally discontinuous. (b) 3D VLAs embed 3D positions into visual representations to improve spatial precision and smoothness of actions, but lack temporal coherence. (c) Our VLA-4D integrates both 3D positions and 1D time into visual representations, and extends the action representation into the spatiotemporal domain to achieve spatiotemporal coherence.
• Figure 2: Our VLA-4D consists of two stages: 1) 4D-aware visual representation. Encode 3D positions and 1D time into 4D spatiotemporal embeddings, and fuse them into visual features via a cross-attention mechanism. 2) Spatiotemporal action representation. Extend action parameters into the spatiotemporal domain, and align multimodal representations into the LLM for robotic action prediction.
• Figure 3: Effect of different visual representations. 3D spatial information enhances the understanding of scene geometry and subsequent action localization, while 1D temporal information further ensures the dynamic perception and temporal action state.
• Figure 4: Illustration of spatiotemporal action representation. Spatial parameters enable fine-grained action planning, while temporal parameters further improve the action coherence during execution.
• Figure 5: Quantitative comparison of VLAs on zero-shot robotic manipulation tasks.