ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulation

TL;DR

ManipBench introduces a large-scale MCQ-based benchmark to evaluate Vision-Language Models on low-level robot manipulation across real-world, fabric, and simulation domains. By extracting keypoints and action cues through a MOKA-style pipeline and testing 33 VLMs from 10 families, it reveals substantial variation in low-level reasoning capabilities and a notable gap to human performance. The study demonstrates a strong, statistically significant link between MCQ-based performance and real-world manipulation success, supporting ManipBench as a practical proxy for embodied robotic proficiency. These findings suggest both the promise of current VLMs for low-level manipulation and the clear need for further advances in grounding and affordance reasoning for robotics.

Abstract

Vision-Language Models (VLMs) have revolutionized artificial intelligence and robotics due to their commonsense reasoning capabilities. In robotic manipulation, VLMs are used primarily as high-level planners, but recent work has also studied their lower-level reasoning ability, which refers to making decisions about precise robot movements. However, the community currently lacks a clear and common benchmark that can evaluate how well VLMs can aid low-level reasoning in robotics. Consequently, we propose a novel benchmark, ManipBench, to evaluate the low-level robot manipulation reasoning capabilities of VLMs across various dimensions, including how well they understand object-object interactions and deformable object manipulation. We extensively test 33 representative VLMs across 10 model families on our benchmark, including variants to test different model sizes. Our evaluation shows that the performance of VLMs significantly varies across tasks, and there is a strong correlation between this performance and trends in our real-world manipulation tasks. It also shows that there remains a significant gap between these models and human-level understanding. See our website at: https://manipbench.github.io.

Figures (30)

• Figure 1: ManipBench is a novel benchmark with over 12,000 multiple-choice questions across three different categories to evaluate low-level physical reasoning capabilities of VLMs in the context of robotic manipulation. For the first question (from real robot data), we do not display the text of all options due to space limitations.
• Figure 2: ManipBench uses real and simulated environments, typically pre-processed with a MOKA-style MOKA2024 pipeline to extract key-points and grid annotations for generating VLM-evaluable multiple choice questions.
• Figure 3: The different tasks, with their descriptions and initial states, present in our real-world experiments.
• Figure 4: The percentage accuracies of the VLMs for evaluating the dimensions of Fabric Manipulation, depicted as a Radar Chart, along with the standard deviation of their performances across these dimensions.
• Figure 5: Raw frames (top) are paired with gripper masks (bottom), initially annotated manually (see orange box) and then propagated using SAM2 ravi2024sam2. Bounding boxes derived from the masks are used for fine-tuning Grounding DINO liu2024grounding.