Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations

TL;DR

This work introduces Robotic Steering, a mechanistic interpretability-based method for finetuning Vision-Language-Action models in robotics. By identifying task-relevant attention heads from few-shot demonstrations and selectively finetuning them with LoRA, the approach achieves comparable or better task success than full-head LoRA while drastically reducing parameters and computation. On-robot experiments with a Franka Emika Panda across five tasks demonstrate improved robustness to environmental changes and strong generalization, including unseen tasks. The method also provides interpretable head activations, linking specific heads to physical task requirements and shaping future directions in task-specific, efficient model adaptation for robotics.

Abstract

Vision-Language Action (VLAs) models promise to extend the remarkable success of vision-language models (VLMs) to robotics. Yet, unlike VLMs in the vision-language domain, VLAs for robotics require finetuning to contend with varying physical factors like robot embodiment, environment characteristics, and spatial relationships of each task. Existing fine-tuning methods lack specificity, adapting the same set of parameters regardless of a task's visual, linguistic, and physical characteristics. Inspired by functional specificity in neuroscience, we hypothesize that it is more effective to finetune sparse model representations specific to a given task. In this work, we introduce Robotic Steering, a finetuning approach grounded in mechanistic interpretability that leverages few-shot demonstrations to identify and selectively finetune task-specific attention heads aligned with the physical, visual, and linguistic requirements of robotic tasks. Through comprehensive on-robot evaluations with a Franka Emika robot arm, we demonstrate that Robotic Steering outperforms LoRA while achieving superior robustness under task variation, reduced computational cost, and enhanced interpretability for adapting VLAs to diverse robotic tasks.

Figures (12)

• Figure 1: Robotic Steering. (left) leverages few-shot examples that capture the inherent physical variability (top) of robotic tasks to identify and selectively finetune only task-relevant attention heads. In contrast, standard approaches (right) specify tasks only with a language expression and update all parameters. Because of this, our novel method is a more performant, efficient, and generalizable approach for few-shot VLA finetuning.
• Figure 2: Robotic Steering Approach. Robotic Steering enables targeted adaptation of VLAs by (1) using few-shot demonstrations to extract task-relevant attention heads, (2) finetuning only these components, (3) and using the sparsely finetuned weights for task inference.
• Figure 3: Scaling Experiments. For the Place Marker in Mug task, we show the (a) success rate versus number of demonstrations and (b) success rate versus number of training iterations for Robotic Steering and full-head LoRA
• Figure 4: Task and Attention Head Selection Visual. We show the selected heads and task visuals for Robotic Steering on the Place Marker in Mug task (top) and the Place Cube in Bowl task (bottom).
• Figure 5: Scaling Experiments. For the Place Marker in Mug task, we show the success rate versus number of heads selected for fine-tuning.