IVRA: Improving Visual-Token Relations for Robot Action Policy with Training-Free Hint-Based Guidance

TL;DR

IVRA addresses the loss of 2D spatial structure when Vision-Language-Action models flatten image patches into 1D token sequences. It is a training-free, inference-time method that extracts an affinity map from the frozen vision encoder and injects affinity-guided pooling into selected layers of the language model, reweighting visual tokens to preserve instance-level spatial cues. Across 2D and 3D benchmarks (VIMA and LIBERO) and real-world tasks, IVRA consistently improves baseline VLA models (LLaRA, OpenVLA, FLOWER) without retraining, including notable gains in challenging, fine-grained manipulation tasks and near-saturation performance regimes. The results demonstrate practical improvements in spatial grounding and manipulation, with minimal computational overhead and a public code release to enable reproducibility and integration into diverse robotic policies.

Abstract

Many Vision-Language-Action (VLA) models flatten image patches into a 1D token sequence, weakening the 2D spatial cues needed for precise manipulation. We introduce IVRA, a lightweight, training-free method that improves spatial understanding by exploiting affinity hints already available in the model's built-in vision encoder, without requiring any external encoder or retraining. IVRA selectively injects these affinity signals into a language-model layer in which instance-level features reside. This inference-time intervention realigns visual-token interactions and better preserves geometric structure while keeping all model parameters fixed. We demonstrate the generality of IVRA by applying it to diverse VLA architectures (LLaRA, OpenVLA, and FLOWER) across simulated benchmarks spanning both 2D and 3D manipulation (VIMA and LIBERO) and on various real-robot tasks. On 2D VIMA, IVRA improves average success by +4.2% over the baseline LLaRA in a low-data regime. On 3D LIBERO, it yields consistent gains over the OpenVLA and FLOWER baselines, including improvements when baseline accuracy is near saturation (96.3% to 97.1%). All code and models will be released publicly. Visualizations are available at: jongwoopark7978.github.io/IVRA

Figures (7)

• Figure 1: Main Overview of Our Method. (a) Left: A frozen vision encoder provides an affinity hint that guides token mixing with weighted pooled tokens, preserving instance-level cues and improving manipulation policy quality. Brighter regions indicate higher affinity relative to the reference point (red dot). (b) Right: Affinity maps after applying IVRA shows sharper object boundaries and clearer object separation, aiding precise robot manipulation.
• Figure 2: Experimental setup with a gripper-equipped arm and overhead RGB camera.
• Figure 3: Visualization of Real-world Scenarios: The top row shows the initial scene and the bottom row the corresponding successful end state for each of the three tasks. Additional columns include trials with distracting objects in the workspace.
• Figure 4: Target Object Task (T1) Visualization. Prompt: "Pick up the orange and drop it into a pan." Both LLaRA+IVRA (top row) and LLaRA (bottom row) select the correct object.
• Figure 5: Object Color Matching Task (T2) Visualization. Prompt: "Pick up object same color as the duck and drop it into a pan." Top row: LLaRA+IVRA correctly identifies and picks up the green broccoli (matching the duck’s color). Bottom row: LLaRA fails to pick up the broccoli.