MAPS: Preserving Vision-Language Representations via Module-Wise Proximity Scheduling for Better Vision-Language-Action Generalization

TL;DR

MAPS tackles the generalization gap in Vision-Language-Action models by introducing Module-Wise Proximity Scheduling, a lightweight, architecture-aware robust fine-tuning framework. It replaces a single global regularization strength with a linear, layer-wise decay of proximity to pretrained weights, tightly anchoring early visual priors (DINOv2) while progressively allowing higher-level language components to adapt to action grounding. Through extensive experiments on SimplerEnv, CALVIN, LIBERO, and real-world Franka Panda tasks across multiple backbones, MAPS delivers consistent ID and out-of-distribution (OOD) gains, sometimes up to +30% in challenging settings, without adding data or parameters. The approach emphasizes empirically discovered hierarchies in module importance and demonstrates that guided proximity to pretrained Vision-Language Models is a simple yet powerful principle for scalable VLA generalization.

Abstract

Vision-Language-Action (VLA) models inherit strong priors from pretrained Vision-Language Models (VLMs), but naive fine-tuning often disrupts these representations and harms generalization. Existing fixes -- freezing modules or applying uniform regularization -- either overconstrain adaptation or ignore the differing roles of VLA components. We present MAPS (Module-Wise Proximity Scheduling), the first robust fine-tuning framework for VLAs. Through systematic analysis, we uncover an empirical order in which proximity constraints should be relaxed to balance stability and flexibility. MAPS linearly schedules this relaxation, enabling visual encoders to stay close to their pretrained priors while action-oriented language layers adapt more freely. MAPS introduces no additional parameters or data, and can be seamlessly integrated into existing VLAs. Across MiniVLA-VQ, MiniVLA-OFT, OpenVLA-OFT, and challenging benchmarks such as SimplerEnv, CALVIN, LIBERO, as well as real-world evaluations on the Franka Emika Panda platform, MAPS consistently boosts both in-distribution and out-of-distribution performance (up to +30%). Our findings highlight empirically guided proximity to pretrained VLMs as a simple yet powerful principle for preserving broad generalization in VLM-to-VLA transfer.

Figures (10)

• Figure 1: Module-Wise Proximity Scheduling (MAPS). MAPS is applied on pretrained VLM components during the action finetuning stage. MAPS (blue dash line) enforces strong preservation on early vision layers while progressively relaxing constraints toward higher-level language layers. In contrast, vanilla finetuning (green dash line) distorts the VLM representation completely away from its pretrained weights (black solid line), and uniform SPD (orange dash/dot line) applies the same constraint everywhere.
• Figure 2: Qualitative comparison of freezing configurations for LIBERO-90 task "put the bowl on the plate" (left to right: full fine-tuning (FFT), freeze VLM, freeze language, freeze vision). FFT fails most, targetting the cabinet instead of the bowl. Freezing VLM/language/vision preserves 2D localization of the bowl but impairs depth reasoning. When vision is frozen, the policy can grasp the bowl but fails to accurately place it on the plate.
• Figure 3: We calculate the $\ell_2$ distance between fine-tuned and pre-trained weights. MAPS produces a smooth, module-aware decay in deviation from pretrained initialization (top). Applying robust finetuning only to the vision stack and full finetuning on language (RFT-V+FFT-L) constrains DINOv2 heavily but leaves SigLIP and language relatively unconstrained (middle). Uniform $\lambda$ (RFT) constrains all layers equally (bottom).
• Figure 4: SimplerEnv qualitative example (new object: Red Bull can). Top: Full fine-tuning fails to complete the task. Bottom: MAPS successfully places the can on the plate.
• Figure 5: CALVIN Results. MAPS improves success rates across all action horizons (25% for MiniVLA-OFT and 3% for OpenVLA-OFT) and average length (+0.7 for MiniVLA-OFT and +0.1 for OpenVLA-OFT).