ManipGPT: Is Affordance Segmentation by Large Vision Models Enough for Articulated Object Manipulation?

TL;DR

ManipGPT provides a vision-driven approach to articulated object manipulation that eliminates iterative sampling and heavy perception pipelines by fine-tuning a large vision model on a small, mixed sim-real dataset and using in-context prompts to generate part-level affordance masks from a single RGB image. The framework decomposes into an Affordance Predictor and an Action Proposer, translating mask information into a contact point, manipulation direction, and impedance-controlled execution guided by surface normals and gradient filtering. Across simulation and real-world tests, ManipGPT achieves strong segmentation accuracy and manipulation success, outperforming several baselines and demonstrating notable generalization to unseen instances, while acknowledging limitations with small, transparent, or boundary-ambiguous objects. The results suggest a practical path toward data-efficient, robust manipulation for articulated objects, with potential extensions to other grippers and grasping strategies and a clear benefit from incorporating real-world data and temporal cues.

Abstract

Visual actionable affordance has emerged as a transformative approach in robotics, focusing on perceiving interaction areas prior to manipulation. Traditional methods rely on pixel sampling to identify successful interaction samples or processing pointclouds for affordance mapping. However, these approaches are computationally intensive and struggle to adapt to diverse and dynamic environments. This paper introduces ManipGPT, a framework designed to predict optimal interaction areas for articulated objects using a large pre-trained vision transformer (ViT). We create a dataset of 9.9k simulated and real images to bridge the visual sim-to-real gap and enhance real-world applicability. By fine-tuning the vision transformer on this small dataset, we significantly improve part-level affordance segmentation, adapting the model's in-context segmentation capabilities to robot manipulation scenarios. This enables effective manipulation across simulated and real-world environments by generating part-level affordance masks, paired with an impedance adaptation policy, sufficiently eliminating the need for complex datasets or perception systems.

Figures (6)

• Figure 1: ManipGPT processes an RGB image with a visual prompt to generate an affordance mask, which determines the contact point and manipulation direction.
• Figure 2: The pipeline is divided into two main modules: Affordance Predictor and Action Proposer. (a) The RGB image and category-specific prompt are input to generate an affordance mask. (b) The affordance mask highlights actionable parts of the object. (c) A surface normal map is used to understand the object's surface orientation. (d) A vector gradient filter refines the normal map by filtering out non-ideal areas. (e) The optimal contact point and manipulation direction are identified. (f) The manipulation direction is set for the gripper's action. (g) An impedance control algorithm guides the robot’s manipulation based on physical feedback.
• Figure 3: Sample RGB images and ground truth affordance masks: real-world examples (left) and simulated images (right). White regions indicate manipulable affordance areas.
• Figure 4: Thermal map comparison of ManipGPT and other models. LISA struggles to mask out all actionable parts for certain categories, such as doors. Other models are lack of generalization ability.
• Figure 5: We test 11 objects in our real-world experiment, numbered 1–11 as listed in Table \ref{['tab:pushing_and_pulling']}. Objects 1–6 belong to the pushing category, while 7–11 are in the pushing category.