Clutter-Resistant Vision-Language-Action Models through Object-Centric and Geometry Grounding

TL;DR

This paper tackles the brittleness of language-conditioned robotic manipulation in clutter by decoupling perception from action. It introduces OBEYED-VLA, which adds a frozen perception grounding module that uses VLM-based object-centric reasoning and depth-based geometry to produce clutter-suppressed views for a pretrained Vision-Language-Action policy, which is fine-tuned only on clean single-object demonstrations. Across real-world UR10e tabletop experiments, OBEYED-VLA shows superior robustness to distractors, absent-target instructions, background changes, and unseen objects, with ablations confirming the critical roles of two-stage object grounding and explicit geometry grounding. The approach improves grounding reliability without synthetic clutter data or perceptual losses during VLA training, offering a practical path to generalizable, clutter-resistant visuomotor control.

Abstract

Recent Vision-Language-Action (VLA) models have made impressive progress toward general-purpose robotic manipulation by post-training large Vision-Language Models (VLMs) for action prediction. Yet most VLAs entangle perception and control in a monolithic pipeline optimized purely for action, which can erode language-conditioned grounding. In our real-world tabletop tests, policies over-grasp when the target is absent, are distracted by clutter, and overfit to background appearance. To address these issues, we propose OBEYED-VLA (OBject-centric and gEometrY groundED VLA), a framework that explicitly disentangles perceptual grounding from action reasoning. Instead of operating directly on raw RGB, OBEYED-VLA augments VLAs with a perception module that grounds multi-view inputs into task-conditioned, object-centric, and geometry-aware observations. This module includes a VLM-based object-centric grounding stage that selects task-relevant object regions across camera views, along with a complementary geometric grounding stage that emphasizes the 3D structure of these objects over their appearance. The resulting grounded views are then fed to a pretrained VLA policy, which we fine-tune exclusively on single-object demonstrations collected without environmental clutter or non-target objects. On a real-world UR10e tabletop setup, OBEYED-VLA substantially improves robustness over strong VLA baselines across four challenging regimes and multiple difficulty levels: distractor objects, absent-target rejection, background appearance changes, and cluttered manipulation of unseen objects. Ablation studies confirm that both semantic grounding and geometry-aware grounding are critical to these gains. Overall, the results indicate that making perception an explicit, object-centric component is an effective way to strengthen and generalize VLA-based robotic manipulation.

Figures (11)

• Figure 1: Perception-grounded visuomotor manipulation in real-world cluttered scenes.(a) Real-world scenarios that stress language-conditioned grounding, including mismatched task queries (absent targets), distractor objects, background appearance shifts, and unseen objects. (b) Typical failure modes of state-of-the-art VLAs, which suffer degraded visual grounding, neglect task instructions, and are brittle to visual distractions, leading to spurious grasps, collisions, or picking incorrect targets. (c) Proposed OBject-centric and gEometrY groundED VLA (OBEYED-VLA) framework: a VLM-driven perceptual module transforms raw RGB observations into task-conditioned, object- and geometry-focused views, enabling the downstream VLA to (i) remain reliable in cluttered scenes (e.g., with multiple distractor objects or shifted backgrounds), (ii) reject infeasible or inconsistent commands and ignore distractors (e.g., absent-target instructions), and (iii) generalize to novel target objects unseen during training, without synthetic clutter data or auxiliary training losses.
• Figure 2: Absent-target sanity check of vision-language grounding. We report pick-up rate (%) for each (requested, shown) object pair, computed over 20 rollouts for all combinations of requested (rows) and shown (columns) objects. Object labels are Ketchup, Mustard, Coffee (coffee bag), and Olive (olive oil bottle), so off-diagonal intensities directly reveal how often the policy grasps when the requested object is absent.
• Figure 3: An overview of OBEYED-VLA architecture. Raw RGB images from base and wrist cameras are first passed through a segmentation network to obtain object-level masks. VLM-based object-centric grounding module then selects a subset of masks corresponding to task-relevant objects, while geometric grounding module applies depth estimation to these masks to produce clutter-suppressed, geometry-aware observations focused on those regions. The resulting perceptually grounded observations, together with the language instruction and robot proprioception, are then fed into a pretrained VLA model that outputs action trajectories; only the VLA is needed to be fine-tuned for downstream tasks, while the perception modules remain frozen to enable plug-and-play integration with different VLAs.
• Figure 4: Object-Centric Grounding Module. The module operates in two stages. First, the VLM parses the task instruction to extract task-relevant objects and, using set-of-mark prompting on the base-view segmentation masks to select the regions corresponding to those objects. We crop the selected base-view regions to produce object-centric reference views and provide these, together with set-of-mark augmented wrist-view image, in a single prompt to the VLM, which predicts the matching wrist regions. The resulting task-relevant masks in both base and wrist views define semantically grounded regions that eliminates distractions and background, isolating only the visual content most relevant to the task instruction.
• Figure 5: Experimental setting: a UR10e robot with parallel jaw gripper and base/wrist cameras. Policies are trained on single-object pick-and-place demonstrations over eight grocery objects. For evaluation, we test both cluttered scenes built from these training categories and generalization by seven additional object categories that are excluded from training.