VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models

TL;DR

This work introduces VLM4VLA, a minimalist, fair pipeline to convert general Vision-Language Models (VLMs) into Vision-Language-Action (VLA) policies with a tiny parameter footprint. Through over 100 experiments across Calvin, SimplerEnv, and Libero benchmarks, it shows that a VLM’s general capabilities do not reliably predict downstream embodied control performance and that the vision encoder is the primary bottleneck. Fine-tuning auxiliary embodied tasks rarely translates into better VLA control, while freezing the vision module generally harms performance; injecting control-relevant signals into the vision encoder yields gains, highlighting a persistent mismatch between VLM pretraining objectives and embodied action planning. The results call for aligning visual representations with low-level control needs and suggest that VLM pretraining is necessary but not sufficient for effective embodied agents.

Abstract

Vision-Language-Action (VLA) models, which integrate pretrained large Vision-Language Models (VLM) into their policy backbone, are gaining significant attention for their promising generalization capabilities. This paper revisits a fundamental yet seldom systematically studied question: how VLM choice and competence translate to downstream VLA policies performance? We introduce VLM4VLA, a minimal adaptation pipeline that converts general-purpose VLMs into VLA policies using only a small set of new learnable parameters for fair and efficient comparison. Despite its simplicity, VLM4VLA proves surprisingly competitive with more sophisticated network designs. Through extensive empirical studies on various downstream tasks across three benchmarks, we find that while VLM initialization offers a consistent benefit over training from scratch, a VLM's general capabilities are poor predictors of its downstream task performance. This challenges common assumptions, indicating that standard VLM competence is necessary but insufficient for effective embodied control. We further investigate the impact of specific embodied capabilities by fine-tuning VLMs on seven auxiliary embodied tasks (e.g., embodied QA, visual pointing, depth estimation). Contrary to intuition, improving a VLM's performance on specific embodied skills does not guarantee better downstream control performance. Finally, modality-level ablations identify the visual module in VLM, rather than the language component, as the primary performance bottleneck. We demonstrate that injecting control-relevant supervision into the vision encoder of the VLM yields consistent gains, even when the encoder remains frozen during downstream fine-tuning. This isolates a persistent domain gap between current VLM pretraining objectives and the requirements of embodied action-planning.

Figures (11)

• Figure 1: An overview of our VLM4VLA framework. (Left) The evaluation pipeline for testing different VLM backbones, which are evaluated on downstream tasks after an optional fine-tuning stage on auxiliary embodied tasks. (Bottom Right) We systematically investigate three factors influencing VLM-to-VLA transfer: the choice of VLM backbone, the impact of fine-tuning on auxiliary embodied tasks, and the influence of different training strategies (frozen vs. fine-tuned different VLM modules, training from scratch). (Top Right) A visualization of inconsistent performance of various VLM backbones across downstream tasks.
• Figure 2: VLA Network in VLM4VLA
• Figure 3: Comparison of the linear relationship between general VLM capabilities and VLA performance.
• Figure 4: Performance of different auxiliary VLM finetune tasks. The Total Length dimension is scaled by a factor of 5 to normalize it to the range [0, 1]. The results for the VLAs trained under each task and for each gradient steps (10k, 15k, 20k, 25k and 30k) are rendered as box plots to provide a view of the impact of different tasks on the VLA's performance and stability.
• Figure 5: During learning, VLMs and VLAs initially follow the same trajectory. At a certain timestep, they diverge into different regions that cause the gap.