RealVLG-R1: A Large-Scale Real-World Visual-Language Grounding Benchmark for Robotic Perception and Manipulation

Abstract

Visual-language grounding aims to establish semantic correspondences between natural language and visual entities, enabling models to accurately identify and localize target objects based on textual instructions. Existing VLG approaches focus on coarse-grained, object-level localization, while traditional robotic grasping methods rely predominantly on geometric cues and lack language guidance, which limits their applicability in language-driven manipulation scenarios. To address these limitations, we propose the RealVLG framework, which integrates the RealVLG-11B dataset and the RealVLG-R1 model to unify real-world visual-language grounding and grasping tasks. RealVLG-11B dataset provides multi-granularity annotations including bounding boxes, segmentation masks, grasp poses, contact points, and human-verified fine-grained language descriptions, covering approximately 165,000 images, over 800 object instances, 1.3 million segmentation, detection, and language annotations, and roughly 11 billion grasping examples. Building on this dataset, RealVLG-R1 employs Reinforcement Fine-tuning on pretrained large-scale vision-language models to predict bounding boxes, segmentation masks, grasp poses, and contact points in a unified manner given natural language instructions. Experimental results demonstrate that RealVLG supports zero-shot perception and manipulation in real-world unseen environments, establishing a unified semantic-visual multimodal benchmark that provides a comprehensive data and evaluation platform for language-driven robotic perception and grasping policy learning. All data and code are publicly available at https://github.com/lif314/RealVLG-R1.

Figures (11)

• Figure 1: We propose RealVLG, a unified framework that integrates the RealVLG-11B dataset and RealVLG-R1 model to enable multi-granularity, zero-shot robotic visual-language grounding and grasping in real-world scenarios.
• Figure 2: Annotation pipeline of the RealVLG-11B dataset. The pipeline integrates automatic language generation, model-based verification, and manual review to generate high-quality multi-granularity visual and language annotations.
• Figure 3: Framework of RealVLG-R1. RealVLG-R1 fine-tunes pretrained LVLMs via reward-driven RL using task-specific verifiable rewards, enabling adaptive learning and improved generalization over bounding boxes, segmentation, grasp rectangles, and contact points.
• Figure 4: Overview of the RealVLG-R1 deployment for real-world Visual-Language Grasping tasks. RealVLG-R1 produces multi-granularity visual–language outputs, which can be leveraged in two complementary grasping strategies: (a) coarse-grained, object-centric grasping, where segmentation masks or bounding boxes are projected into 3D point clouds to generate 6-DoF grasp poses via a 3D grasping module; (b) fine-grained, part-level grasping, where 2D grasp predictions are directly transformed into executable 6-DoF poses using depth and camera parameters, enabling semantically precise manipulation. This design supports hierarchical control from global geometry to detailed semantic structures.
• Figure 5: Human-Verification System. This application provides an interactive interface for human-in-the-loop verification, allowing users to review, correct, and confirm automatically generated visual-language annotations. It serves as a crucial component for ensuring the quality and reliability of RealVLG-11B dataset annotations.