RoboGround: Robotic Manipulation with Grounded Vision-Language Priors

TL;DR

This work tackles poor generalization in robotic manipulation by introducing grounding masks as a structured intermediate representation derived from a grounded vision-language prior. The RoboGround framework combines a GLaMM-based grounded VLM that outputs target-object and placement-area masks with a Grounded Perceiver-enhanced policy that attends to mask-guided regions, enabling robust zero-shot and cross-domain performance. A large-scale data generation pipeline built on RoboCasa and Objaverse yields $24K$ demonstrations and $112K$ instructions across $176$ object categories, driving diversity in appearance, space, and commonsense reasoning. Experimental results show that mask-guided policies significantly improve generalization to unseen objects and instructions, though grasping remains a nontrivial bottleneck; future work includes integrating grasp-pose predictors and exploring end-to-end architectures for longer-horizon tasks.

Abstract

Recent advancements in robotic manipulation have highlighted the potential of intermediate representations for improving policy generalization. In this work, we explore grounding masks as an effective intermediate representation, balancing two key advantages: (1) effective spatial guidance that specifies target objects and placement areas while also conveying information about object shape and size, and (2) broad generalization potential driven by large-scale vision-language models pretrained on diverse grounding datasets. We introduce RoboGround, a grounding-aware robotic manipulation system that leverages grounding masks as an intermediate representation to guide policy networks in object manipulation tasks. To further explore and enhance generalization, we propose an automated pipeline for generating large-scale, simulated data with a diverse set of objects and instructions. Extensive experiments show the value of our dataset and the effectiveness of grounding masks as intermediate guidance, significantly enhancing the generalization abilities of robot policies.

Figures (8)

• Figure 1: Examples of generated data and mask guidance for manipulation policy. The generated data includes more object distractors in the scene, leading to higher scene complexity. The robot policy is guided by masks to localize the target object and placement area.
• Figure 2: Data Generation Pipeline. The pipeline is composed of three key stages: (a) First, we extract informative object attributes in both keyword and descriptive phrase formats; (b) Next, appearance-based instructions are generated using these attributes, where keywords filter objects and descriptive phrases calculate appearance similarity; (c) Finally, spatial and commonsense instructions are generated through rule-based methods and GPT-generated techniques, respectively.
• Figure 3: Overall Architecture of RoboGround. To enhance policy generalization, we leverage grounding masks as intermediate representations for spatial guidance. Specifically, (a) The grounded vision-language model processes the instruction and image observation to generate target masks. (b) The grounded policy network integrates mask guidance by concatenating masks with the image input and directing attention within the grounded perceiver.
• Figure 4: Comparison of Word Clouds: Original Data (left) vs. Generated Data (right).
• Figure 5: Prompt for Filtering Kitchen-related Objects.