PhysVLM: Enabling Visual Language Models to Understand Robotic Physical Reachability

TL;DR

This work addresses the gap in vision-language models for embodied robotic tasks caused by a lack of understanding of robotic physical reachability. It introduces Space-Physical Reachability Map (S-P Map), a robot-agnostic representation of reachability, and PhysVLM, a dual-branch vision-language model that fuses S-P Map with visual and textual reasoning through an LLM decoder. A large-scale Phys100K dataset and the EQA-phys benchmark are released to train and evaluate across six robots in simulation and real-world environments, with PhysVLM outperforming GPT-4o and other embodied VLMs on key benchmarks. Experiments demonstrate robust cross-robot generalization and improved task planning when reachability constraints are considered, highlighting potential for safer, more reliable robotic decision-making in industrial and assistive settings.

Abstract

Understanding the environment and a robot's physical reachability is crucial for task execution. While state-of-the-art vision-language models (VLMs) excel in environmental perception, they often generate inaccurate or impractical responses in embodied visual reasoning tasks due to a lack of understanding of robotic physical reachability. To address this issue, we propose a unified representation of physical reachability across diverse robots, i.e., Space-Physical Reachability Map (S-P Map), and PhysVLM, a vision-language model that integrates this reachability information into visual reasoning. Specifically, the S-P Map abstracts a robot's physical reachability into a generalized spatial representation, independent of specific robot configurations, allowing the model to focus on reachability features rather than robot-specific parameters. Subsequently, PhysVLM extends traditional VLM architectures by incorporating an additional feature encoder to process the S-P Map, enabling the model to reason about physical reachability without compromising its general vision-language capabilities. To train and evaluate PhysVLM, we constructed a large-scale multi-robot dataset, Phys100K, and a challenging benchmark, EQA-phys, which includes tasks for six different robots in both simulated and real-world environments. Experimental results demonstrate that PhysVLM outperforms existing models, achieving a 14\% improvement over GPT-4o on EQA-phys and surpassing advanced embodied VLMs such as RoboMamba and SpatialVLM on the RoboVQA-val and OpenEQA benchmarks. Additionally, the S-P Map shows strong compatibility with various VLMs, and its integration into GPT-4o-mini yields a 7.1\% performance improvement.

Figures (4)

• Figure 1: Existing VLM models, such as GPT-4o, may generate inaccurate or impractical responses due to poor comprehension of robotic physical reachability. The proposed PhysVLM integrates vision-language capabilities with an understanding of robotic physical reachability.
• Figure 2: Overview of PhysVLM. Starting with robot parameters and an egocentric depth map, an S-P Map is generated via unified physical reachability encoding. Using the S-P Map, image, and instruction text, PhysVLM generates textual output considering the robot's physical reachability.
• Figure 3: Details of the Phys100K Dataset and EQA-Phys Benchmark.
• Figure 4: Visual comparison of PhysVLM (ours), GPT-4o, and SpatialBot.