RobotSeg: A Model and Dataset for Segmenting Robots in Image and Video

TL;DR

RobotSeg introduces a dedicated, structure-aware foundation model for segmenting robots in both images and videos, addressing SAM 2’s limitations with a structure-enhanced memory associator, a robot prompt generator, and a label-efficient training strategy. It also provides the Video Robot Segmentation (VRS) dataset, the first large-scale video benchmark for robot segmentation across multiple embodiments and environments. Extensive experiments show state-of-the-art performance, improved temporal coherence, and robust generalization to diverse robots while maintaining efficiency. The work enables reliable robot perception for downstream tasks such as data augmentation, sim-to-real transfer, and safety monitoring.

Abstract

Accurate robot segmentation is a fundamental capability for robotic perception. It enables precise visual servoing for VLA systems, scalable robot-centric data augmentation, accurate real-to-sim transfer, and reliable safety monitoring in dynamic human-robot environments. Despite the strong capabilities of modern segmentation models, surprisingly it remains challenging to segment robots. This is due to robot embodiment diversity, appearance ambiguity, structural complexity, and rapid shape changes. Embracing these challenges, we introduce RobotSeg, a foundation model for robot segmentation in image and video. RobotSeg is built upon the versatile SAM 2 foundation model but addresses its three limitations for robot segmentation, namely the lack of adaptation to articulated robots, reliance on manual prompts, and the need for per-frame training mask annotations, by introducing a structure-enhanced memory associator, a robot prompt generator, and a label-efficient training strategy. These innovations collectively enable a structure-aware, automatic, and label-efficient solution. We further construct the video robot segmentation (VRS) dataset comprising over 2.8k videos (138k frames) with diverse robot embodiments and environments. Extensive experiments demonstrate that RobotSeg achieves state-of-the-art performance on both images and videos, establishing a strong foundation for future advances in robot perception.

Figures (20)

• Figure 1: Although existing state-of-the-art segmentation models (e.g., RoboEngine yuan2025roboengine and SAM 2.1 sam2) are highly capable, surprisingly they struggle to segment robots: they fail to cope with diverse embodiments (1-3 columns), often confuse robots with cluttered backgrounds (4th column), break down when facing complex structures (5th column), and fail to handle rapid shape changes (6th-7th columns). In contrast, our RobotSeg model (last row) achieves robust robot segmentation across diverse embodiments and scenes, and further supports user-provided prompts (e.g., clicks or bounding boxes) to refine the segmentation results (last column).
• Figure 2: Examples from our VRS dataset. It encompasses different scenes, lighting conditions, and robot embodiments. Each example shows the RGB sequence (top) and robot annotation masks (bottom), where the robot arm is highlighted in red and the gripper in green.
• Figure 3: Robot embodiments in our video robot segmentation (VRS) dataset.
• Figure 4: Overview of our RobotSeg. Building upon SAM 2 sam2, it introduces a structure-enhanced memory associator, a robot prompt generator, and a label-efficient training strategy to enable structure-aware, automatic, and training-label-efficient robot segmentation.
• Figure 5: Illustration of the structure-enhanced memory associator. It encodes previous features and masks into memory to guide temporal context integration in the top branch and robot boundary perception in the bottom branch, where detected boundaries are used for structure enhancement of the top features.