JEPA-VLA: Video Predictive Embedding is Needed for VLA Models

TL;DR

The work addresses the inefficiency and limited generalization of vision-language-action models by identifying deficiencies in static visual representations for robotics. It shows that video-based predictive embeddings, exemplified by V-JEPA2, capture task-relevant state information and temporal dynamics that serve as effective policy priors, outperforming image-based and language-based pretraining in probing tasks. The authors introduce JEPA-VLA, a flexible fusion framework that either concatenates predictive embeddings (early fusion) or uses gated cross-attention to preserve pretrained priors while integrating new information. Across LIBERO, LIBERO-plus, RoboTwin2.0, and real-world tasks, JEPA-VLA yields consistent improvements in sample efficiency and generalization, demonstrating the practical value of video-predictive pretraining for embodied AI. This work suggests that leveraging video-scale representations is a key step toward more robust and generalizable robotic agents.

Abstract

Recent vision-language-action (VLA) models built upon pretrained vision-language models (VLMs) have achieved significant improvements in robotic manipulation. However, current VLAs still suffer from low sample efficiency and limited generalization. This paper argues that these limitations are closely tied to an overlooked component, pretrained visual representation, which offers insufficient knowledge on both aspects of environment understanding and policy prior. Through an in-depth analysis, we find that commonly used visual representations in VLAs, whether pretrained via language-image contrastive learning or image-based self-supervised learning, remain inadequate at capturing crucial, task-relevant environment information and at inducing effective policy priors, i.e., anticipatory knowledge of how the environment evolves under successful task execution. In contrast, we discover that predictive embeddings pretrained on videos, in particular V-JEPA 2, are adept at flexibly discarding unpredictable environment factors and encoding task-relevant temporal dynamics, thereby effectively compensating for key shortcomings of existing visual representations in VLAs. Building on these observations, we introduce JEPA-VLA, a simple yet effective approach that adaptively integrates predictive embeddings into existing VLAs. Our experiments demonstrate that JEPA-VLA yields substantial performance gains across a range of benchmarks, including LIBERO, LIBERO-plus, RoboTwin2.0, and real-robot tasks.

Figures (5)

• Figure 1: Comparison of visual representations commonly used in VLAs:(a) Image-based self-supervised learning (e.g., the DINO family) yields precise visual representations, but is relatively insensitive to task relevance and preserves task-irrelevant details. (b) Language-image contrastive learning (e.g., CLIP and the SigLIP family) emphasizes instruction-aligned entities and semantics, yet may capture less low-level task-relevant information beyond what is explicitly described in text. (c) Video-based predictive learning (e.g., V-JEPA 2) provides state-centric representations for task-relevant objects while also encoding temporal regularities that act as policy priors, which are difficult to obtain from image-only pretraining.
• Figure 2: Analysis of visual representations for VLAs.(a) Trajectory demonstration and state definition. Given observations $\{o_t\}$, we factorize the underlying state at each timestep as task-relevant and task-irrelevant parts. (b) Experimental setup. We freeze different vision encoders and train a lightweight ViT head to regress or predict environment states. More details can be found in Appendix \ref{['appendix:analysis']}. (c) Experimental results. V-JEPA 2 achieves consistently lower relative loss on task-relevant regression and prediction, compared to DINOv2 and SigLIP, while showing no advantage on task-irrelevant (lighting/background) regression, suggesting that V-JEPA 2 better captures task-relevant environment states and policy priors while discarding nuisance factors.
• Figure 3: Evaluation benchmarks, including (a) LIBERO, (b) LIBERO-plus, (c) RoboTwin, (d) a real-world task, and (e) CortexBench.
• Figure 4: JEPA-VLA fusion architecture. (a) For VLAs without large-scale robotic pretraining, V-JEPA 2 representations are concatenated as additional input embeddings. (b) For VLAs pretrained on large-scale robotic datasets, we integrate V-JEPA 2 representations using gated cross-attention, which enables adaptive fusion while preserving pretrained priors.
• Figure 5: Examples of real-world testing, respectively in standard, modified layouts and lights settings.