Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PhysQuantAgent: An Inference Pipeline of Mass Estimation for Vision-Language Models

Hisayuki Yokomizo, Taiki Miyanishi, Yan Gang, Shuhei Kurita, Nakamasa Inoue, Yusuke Iwasawa

Abstract

Vision-Language Models (VLMs) are increasingly applied to robotic perception and manipulation, yet their ability to infer physical properties required for manipulation remains limited. In particular, estimating the mass of real-world objects is essential for determining appropriate grasp force and ensuring safe interaction. However, current VLMs lack reliable mass reasoning capabilities, and most existing benchmarks do not explicitly evaluate physical quantity estimation under realistic sensing conditions. In this work, we propose PhysQuantAgent, a framework for real-world object mass estimation using VLMs, together with VisPhysQuant, a new benchmark dataset for evaluation. VisPhysQuant consists of RGB-D videos of real objects captured from multiple viewpoints, annotated with precise mass measurements. To improve estimation accuracy, we introduce three visual prompting methods that enhance the input image with object detection, scale estimation, and cross-sectional image generation to help the model comprehend the size and internal structure of the target object. Experiments show that visual prompting significantly improves mass estimation accuracy on real-world data, suggesting the efficacy of integrating spatial reasoning with VLM knowledge for physical inference.

PhysQuantAgent: An Inference Pipeline of Mass Estimation for Vision-Language Models

Abstract

Vision-Language Models (VLMs) are increasingly applied to robotic perception and manipulation, yet their ability to infer physical properties required for manipulation remains limited. In particular, estimating the mass of real-world objects is essential for determining appropriate grasp force and ensuring safe interaction. However, current VLMs lack reliable mass reasoning capabilities, and most existing benchmarks do not explicitly evaluate physical quantity estimation under realistic sensing conditions. In this work, we propose PhysQuantAgent, a framework for real-world object mass estimation using VLMs, together with VisPhysQuant, a new benchmark dataset for evaluation. VisPhysQuant consists of RGB-D videos of real objects captured from multiple viewpoints, annotated with precise mass measurements. To improve estimation accuracy, we introduce three visual prompting methods that enhance the input image with object detection, scale estimation, and cross-sectional image generation to help the model comprehend the size and internal structure of the target object. Experiments show that visual prompting significantly improves mass estimation accuracy on real-world data, suggesting the efficacy of integrating spatial reasoning with VLM knowledge for physical inference.
Paper Structure (16 sections, 2 equations, 13 figures, 2 tables)

This paper contains 16 sections, 2 equations, 13 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. Related Work
  3. Physical Properties Estimation
  4. Physical Understanding Benchmarks
  5. Object Datasets
  6. Method
  7. Task Formulation
  8. PhysQuantAgent
  9. Dataset
  10. Data Collection
  11. Dataset Statistics
  12. Experiment
  13. Experimental Setup
  14. Experimental Results
  15. Robot Application
  16. ...and 1 more sections

Figures (13)

  • Figure 1: Comparison of previous mass estimation methods and our proposed PhysQuantAgent. While prior approaches rely on computationally expensive 3D reconstruction from RGB-D images, PhysQuantAgent directly infers object mass with a VLM, enabling fast estimation for grasp force control.
  • Figure 2: PhysQuantAgent framework: First, the VLM selects the tool to use (Stage1), then estimates the mass of the object from the original image and the image with the tool applied (Stage2).
  • Figure 3: Example of the instruction prompt in PhysQuantAgent.
  • Figure 4: Pinhole camera model.
  • Figure 5: Examples of Visual Prompting.
  • ...and 8 more figures