Stable Language Guidance for Vision-Language-Action Models

TL;DR

Empirical results across diverse manipulation benchmarks demonstrate that RSS achieves state-of-the-art robustness, maintaining performance even under adversarial linguistic perturbations.

Abstract

Vision-Language-Action (VLA) models have demonstrated impressive capabilities in generalized robotic control; however, they remain notoriously brittle to linguistic perturbations. We identify a critical ``modality collapse'' phenomenon where strong visual priors overwhelm sparse linguistic signals, causing agents to overfit to specific instruction phrasings while ignoring the underlying semantic intent. To address this, we propose \textbf{Residual Semantic Steering (RSS)}, a probabilistic framework that disentangles physical affordance from semantic execution. RSS introduces two theoretical innovations: (1) \textbf{Monte Carlo Syntactic Integration}, which approximates the true semantic posterior via dense, LLM-driven distributional expansion, and (2) \textbf{Residual Affordance Steering}, a dual-stream decoding mechanism that explicitly isolates the causal influence of language by subtracting the visual affordance prior. Theoretical analysis suggests that RSS effectively maximizes the mutual information between action and intent while suppressing visual distractors. Empirical results across diverse manipulation benchmarks demonstrate that RSS achieves state-of-the-art robustness, maintaining performance even under adversarial linguistic perturbations.

Figures (11)

• Figure 1: Taxonomy of Language Instruction Perturbations. We identify three distinct failure modes in VLA instruction following: (1) Destructive Instruction Overwriting, where critical semantic tokens are lost or masked (e.g., masking the drawer location); (2) Obfuscated Instruction Reinterpretation, where the model fails to ground synonymous or verbose descriptions (e.g., "beverage container" vs. "mug"); and (3) Out-of-Distribution Semantic Transfer, where the instruction targets a valid but unlearned goal configuration (e.g., placing on a "cabinet" instead of the training-set "stove").
• Figure 2: Overview of Residual Semantic Steering (RSS). To combat instruction blindness, RSS operates in two stages. Left:Monte Carlo Syntactic Integration utilizes an Oracle Teacher to generate a dense linguistic neighborhood around a seed instruction. Optimizing over this distribution forces the policy to learn representations that are invariant to syntactic perturbations. Right:Residual Affordance Steering mitigates visual prior dominance. By subtracting the unconditioned "visual instinct" (Base Affordance) from the standard prediction, we isolate and amplify the residual semantic signal, ensuring the policy follows the specific user intent rather than generic visual attractors.
• Figure 3: Comparison on the LIBERO variant R3-Reasoning Chain. In the "open the top drawer and put the bowl inside" task, our model consistently outperforms the baseline under reasoning-chain–perturbed instructions, demonstrating a stronger ability to follow multi-step semantic constraints and accurately complete the task despite increased linguistic complexity.
• Figure 4: Ablation of steering coefficient and denoising steps on destructive instruction overwriting. Success rates (SR, %) across instruction variants under different steering coefficients for $\pi_0$ (a) and $\pi_{0.5}$ (b), and different denoising steps for $\pi_0$ (c) and $\pi_{0.5}$ (d), illustrating the effect of guidance and generation depth on robustness to instruction perturbations.
• Figure 5: Training loss curves. We report the training loss trajectories of different model variants throughout optimization. RAS: Residual Affordance Steering; MCSI: Monte Carlo Syntactic Integration.