Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models?

Zhihui Yang, Yupei Wang, Kaijie Mo, Zhe Zhao, Renfen Hu

TL;DR

This work probes whether visual grounding improves embodied knowledge understanding in large language models by introducing SensoryVec and PerceptualQA, a benchmark grounded in the psychological perceptual system that covers visual, auditory, tactile, gustatory, olfactory, and interoceptive modalities. Across $9$ perceptual subtasks and over $1,700$ questions, the study evaluates $\sim$ $30$ state-of-the-art models, finding that vision-language models do not outperform text-only LMs and that visual modalities lag relative to other senses, especially in spatial reasoning. Analyses reveal vector representations are biased by surface form and word frequency, limiting the capture of perceptual contrasts; fine-tuning yields modest gains but fails to reach human performance. The results indicate that current multimodal grounding strategies inadequately integrate embodied knowledge, underscoring the need for richer data, interaction-based training, and architectures that better fuse perceptual signals with language to bridge the digital-physical divide.

Abstract

Despite significant progress in multimodal language models (LMs), it remains unclear whether visual grounding enhances their understanding of embodied knowledge compared to text-only models. To address this question, we propose a novel embodied knowledge understanding benchmark based on the perceptual theory from psychology, encompassing visual, auditory, tactile, gustatory, olfactory external senses, and interoception. The benchmark assesses the models' perceptual abilities across different sensory modalities through vector comparison and question-answering tasks with over 1,700 questions. By comparing 30 state-of-the-art LMs, we surprisingly find that vision-language models (VLMs) do not outperform text-only models in either task. Moreover, the models perform significantly worse in the visual dimension compared to other sensory dimensions. Further analysis reveals that the vector representations are easily influenced by word form and frequency, and the models struggle to answer questions involving spatial perception and reasoning. Our findings underscore the need for more effective integration of embodied knowledge in LMs to enhance their understanding of the physical world.

Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models?

TL;DR

This work probes whether visual grounding improves embodied knowledge understanding in large language models by introducing SensoryVec and PerceptualQA, a benchmark grounded in the psychological perceptual system that covers visual, auditory, tactile, gustatory, olfactory, and interoceptive modalities. Across perceptual subtasks and over questions, the study evaluates state-of-the-art models, finding that vision-language models do not outperform text-only LMs and that visual modalities lag relative to other senses, especially in spatial reasoning. Analyses reveal vector representations are biased by surface form and word frequency, limiting the capture of perceptual contrasts; fine-tuning yields modest gains but fails to reach human performance. The results indicate that current multimodal grounding strategies inadequately integrate embodied knowledge, underscoring the need for richer data, interaction-based training, and architectures that better fuse perceptual signals with language to bridge the digital-physical divide.

Abstract

Despite significant progress in multimodal language models (LMs), it remains unclear whether visual grounding enhances their understanding of embodied knowledge compared to text-only models. To address this question, we propose a novel embodied knowledge understanding benchmark based on the perceptual theory from psychology, encompassing visual, auditory, tactile, gustatory, olfactory external senses, and interoception. The benchmark assesses the models' perceptual abilities across different sensory modalities through vector comparison and question-answering tasks with over 1,700 questions. By comparing 30 state-of-the-art LMs, we surprisingly find that vision-language models (VLMs) do not outperform text-only models in either task. Moreover, the models perform significantly worse in the visual dimension compared to other sensory dimensions. Further analysis reveals that the vector representations are easily influenced by word form and frequency, and the models struggle to answer questions involving spatial perception and reasoning. Our findings underscore the need for more effective integration of embodied knowledge in LMs to enhance their understanding of the physical world.
Paper Structure (38 sections, 2 figures, 15 tables)

This paper contains 38 sections, 2 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Comparing VLMs and text-only LMs
  4. Multimodal reasoning benchmarks
  5. Embodied AI
  6. Method
  7. SensoryVec Task
  8. PerceptualQA Task
  9. Experiments
  10. Models and Settings
  11. Results
  12. SensoryVec Task
  13. PerceptualQA Task
  14. Discussion
  15. Conclusion
  16. ...and 23 more sections

Figures (2)

  • Figure 1: The two tasks in our embodied knowledge understanding benchmark.
  • Figure 2: Box plots of accuracy on SensoryVec (a) and PerceptualQA (b) subtasks. PerceptualQA subtasks include Visual-Color Attributes(V-CA), Visual-Colors in Nature(V-CN), Visual-Geometry and Transformations(V-GT), Visual-Symbols(V-S), Visual-Body(V-B), Auditory(A), Tactile(T), Gustatory(G), and Olfactory(O). Red dashed lines indicate human performance.