Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision-Language Model-based Physical Reasoning for Robot Liquid Perception

Wenqiang Lai, Yuan Gao, Tin Lun Lam

TL;DR

This work demonstrates a novel action–reasoning paradigm that grounds a vision–language model (GPT-4V) in physical world feedback to perceive and recognize liquids through interactive multimodal perception. By combining Look and Shake actions with image-based perception and haptic feedback, the approach enables GPT-4V to qualitatively estimate viscosity and perform liquid recognition across varied containers without training. The method achieves 86.0% recognition accuracy, surpassing a vision-only baseline of 69.0%, and highlights the value of integrating physical knowledge into LVLM-driven robotic reasoning. The findings suggest practical implications for deploying LVLMs as embodied reasoning agents and point to future work in validating multiple LVLMs and expanding to additional perceptual modalities.

Abstract

There is a growing interest in applying large language models (LLMs) in robotic tasks, due to their remarkable reasoning ability and extensive knowledge learned from vast training corpora. Grounding LLMs in the physical world remains an open challenge as they can only process textual input. Recent advancements in large vision-language models (LVLMs) have enabled a more comprehensive understanding of the physical world by incorporating visual input, which provides richer contextual information than language alone. In this work, we proposed a novel paradigm that leveraged GPT-4V(ision), the state-of-the-art LVLM by OpenAI, to enable embodied agents to perceive liquid objects via image-based environmental feedback. Specifically, we exploited the physical understanding of GPT-4V to interpret the visual representation (e.g., time-series plot) of non-visual feedback (e.g., F/T sensor data), indirectly enabling multimodal perception beyond vision and language using images as proxies. We evaluated our method using 10 common household liquids with containers of various geometry and material. Without any training or fine-tuning, we demonstrated that our method can enable the robot to indirectly perceive the physical response of liquids and estimate their viscosity. We also showed that by jointly reasoning over the visual and physical attributes learned through interactions, our method could recognize liquid objects in the absence of strong visual cues (e.g., container labels with legible text or symbols), increasing the accuracy from 69.0% -- achieved by the best-performing vision-only variant -- to 86.0%.

Vision-Language Model-based Physical Reasoning for Robot Liquid Perception

TL;DR

This work demonstrates a novel action–reasoning paradigm that grounds a vision–language model (GPT-4V) in physical world feedback to perceive and recognize liquids through interactive multimodal perception. By combining Look and Shake actions with image-based perception and haptic feedback, the approach enables GPT-4V to qualitatively estimate viscosity and perform liquid recognition across varied containers without training. The method achieves 86.0% recognition accuracy, surpassing a vision-only baseline of 69.0%, and highlights the value of integrating physical knowledge into LVLM-driven robotic reasoning. The findings suggest practical implications for deploying LVLMs as embodied reasoning agents and point to future work in validating multiple LVLMs and expanding to additional perceptual modalities.

Abstract

There is a growing interest in applying large language models (LLMs) in robotic tasks, due to their remarkable reasoning ability and extensive knowledge learned from vast training corpora. Grounding LLMs in the physical world remains an open challenge as they can only process textual input. Recent advancements in large vision-language models (LVLMs) have enabled a more comprehensive understanding of the physical world by incorporating visual input, which provides richer contextual information than language alone. In this work, we proposed a novel paradigm that leveraged GPT-4V(ision), the state-of-the-art LVLM by OpenAI, to enable embodied agents to perceive liquid objects via image-based environmental feedback. Specifically, we exploited the physical understanding of GPT-4V to interpret the visual representation (e.g., time-series plot) of non-visual feedback (e.g., F/T sensor data), indirectly enabling multimodal perception beyond vision and language using images as proxies. We evaluated our method using 10 common household liquids with containers of various geometry and material. Without any training or fine-tuning, we demonstrated that our method can enable the robot to indirectly perceive the physical response of liquids and estimate their viscosity. We also showed that by jointly reasoning over the visual and physical attributes learned through interactions, our method could recognize liquid objects in the absence of strong visual cues (e.g., container labels with legible text or symbols), increasing the accuracy from 69.0% -- achieved by the best-performing vision-only variant -- to 86.0%.
Paper Structure (17 sections, 2 equations, 7 figures, 2 tables)

This paper contains 17 sections, 2 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Models for Physical Reasoning
  4. Robot Liquid Perception
  5. Methodology
  6. System Architecture
  7. Multimodal Interactive Perception
  8. Vision
  9. Haptics
  10. Experiments
  11. Hardware
  12. Data Collection
  13. Relative Viscosity Estimation via Haptics
  14. Liquid Recognition via Multimodal Feedback
  15. Case Studies
  16. ...and 2 more sections

Figures (7)

  • Figure 1: Our method firstly predicts the visual and physical properties of the queried liquid object using commonsense knowledge (bolded text), and then explores liquids via Look and Shake actions interactively to estimate their properties from image-based feedback. Finally, the liquid with most consistent properties is selected as the answer.
  • Figure 2: To recognize a target liquid object, we prompt (yellow) GPT-4V to reason, act and perceive in a closed-loop. It first predicts the properties of the target liquid object using prior knowledge, and selects an appropriate action to perceive the actual properties from haptic (blue) and visual (pink) feedback. Haptic feedback is denoised and converted into time-series plot, while visual feedback is pre-processed to include object bounding boxes. Based on the predicted and perceived properties, GPT-4V plans for next action to gather more information. GPT-4V terminates the loop with the action Finish when the information is sufficient to recognize the target object. Dashed arrows represent potential action-reasoning paths not selected in the current loop.
  • Figure 3: Ten common household liquids evaluated in the experiments placed in a line on a table, each surrounded by a bounding boxed indexed between 0 and 9 from left to right: coke, water, olive oil, peanut oil, soy sauce, whiskey, balsamic vinegar, orange juice, honey, milk. (Top) Liquids in their original packaging with text and symbols on the labels. (Bottom) Same liquids with labels being removed or rendered invisible.
  • Figure 4: The plots of 10-second F/T sensor signals collected after shaking the containers of 10 common household liquids filled at one third, half, and two thirds of capacity of their containers, sorted by their viscosity. Signals were filtered using a 5th order low-pass with a cut-off of 2 Hz and standardized.
  • Figure 5: The error breakdown of plain and knowledge-enhanced prompts the absolute ground truth viscosity difference of liquid pairs.
  • ...and 2 more figures