Don't Blind Your VLA: Aligning Visual Representations for OOD Generalization

TL;DR

This work reveals that fine-tuning Vision-Language-Action models on robotic tasks can erode inherited vision-language grounding, causing attention misalignment and representation collapse. It introduces VL-Think, a diagnostic suite to assess VL knowledge transfer from large vision-language models to embodied agents, and proposes Visual Representation Alignment to anchor VLA visual features to a frozen vision teacher. Across Simpler-based tasks and OpenVLA variants, the alignment method yields consistent improvements in OOD generalization, linear probing, and qualitative attention, while simple freezing fails. The findings highlight the importance of maintaining cross-modal semantic structure during fine-tuning and provide a scalable, energy-efficient method to preserve VL capabilities in embodied AI systems.

Abstract

The growing success of Vision-Language-Action (VLA) models stems from the promise that pretrained Vision-Language Models (VLMs) can endow agents with transferable world knowledge and vision-language (VL) grounding, laying a foundation for action models with broader generalization. Yet when these VLMs are adapted to the action modality, it remains unclear to what extent their original VL representations and knowledge are preserved. In this work, we conduct a systematic study of representation retention during VLA fine-tuning, showing that naive action fine-tuning leads to degradation of visual representations. To characterize and measure these effects, we probe VLA's hidden representations and analyze attention maps, further, we design a set of targeted tasks and methods that contrast VLA models with their counterpart VLMs, isolating changes in VL capabilities induced by action fine-tuning. We further evaluate a range of strategies for aligning visual representations and introduce a simple yet effective method that mitigates degradation and yields improved generalization to out-of-distribution (OOD) scenarios. Taken together, our analysis clarifies the trade-off between action fine-tuning and the degradation of VL representations and highlights practical approaches to recover inherited VL capabilities. Code is publicly available: https://blind-vla-paper.github.io

Figures (6)

• Figure 1: Visual alignment method overview. Mid-level VLA features are projected onto a normalized sphere and aligned with teacher embeddings, preserving visual semantics and improving OOD generalization. Bottom plots show comparison with standard SFT across three generalization axes on the Simpler-based benchmark liu2025rl4vla.
• Figure 2: Overview of the proposed method. (a, b) Training pipeline with visual alignment loss -- no extra overhead, only precomputed teacher features and a lightweight regularization term during SFT. (c) Conceptual illustration of the loss landscape for VL tasks: the core idea is to optimize the model with respect to the action objective while preserving performance on VL understanding.
• Figure 3: VL-Think Task Suite examples. Each panel illustrates a pick-and-place episode where the agent must place an object on the board matching the instructed concept (e.g., color, number, symbol, or category).
• Figure 4: Attention map comparison: the strongest and most semantically grounded attention appears around middle layers. OpenVLA fine-tuned with our proposed method (OpenVLA Align) maintains object-aligned focus in attention maps, while default OpenVLA SFT shows diffused and noisy patterns, indicating loss of visual-language grounding (for more results see Appendix \ref{['fig:do_you_see_canv2']}).
• Figure 5: t-SNE visualization of token embeddings for Qwen2.5-VL, PrismaticVLM, and OpenVLA. While PrismaticVLM and Qwen2.5-VL maintains well-separated clusters for target objects, OpenVLA shows huge overlap across classes, indicating that action fine-tuning causes representations collapse.