Grounded Concreteness: Human-Like Concreteness Sensitivity in Vision-Language Models

TL;DR

This paper investigates whether vision-language models (VLMs) develop more human-like concreteness sensitivity than text-only LLMs when evaluated with text prompts, using matched Llama backbones and treating multimodal pretraining as an ablation on perceptual grounding. The authors triangulate evidence across three levels—output QA performance as a function of question concreteness, embedding geometry showing concreteness-based clustering, and attention dynamics via entropy measures—plus token-level concreteness judgments aligned to human norms. Across two model scales, VLMs show larger gains on concrete questions, tighter concreteness-centered embedding structure, lower attention entropy for concrete tokens, and closer human-aligned concreteness ratings, suggesting grounding signals enrich concrete semantics. The findings support grounded cognition theories and provide a principled, interpretable axis for comparing model families, with implications for diagnostic interpretability and the design of future multimodal systems. They also acknowledge limitations, including generalization beyond a single model family and potential token-frequency confounds, pointing to directions for developmental analyses and broader cross-model validation.

Abstract

Do vision--language models (VLMs) develop more human-like sensitivity to linguistic concreteness than text-only large language models (LLMs) when both are evaluated with text-only prompts? We study this question with a controlled comparison between matched Llama text backbones and their Llama Vision counterparts across multiple model scales, treating multimodal pretraining as an ablation on perceptual grounding rather than access to images at inference. We measure concreteness effects at three complementary levels: (i) output behavior, by relating question-level concreteness to QA accuracy; (ii) embedding geometry, by testing whether representations organize along a concreteness axis; and (iii) attention dynamics, by quantifying context reliance via attention-entropy measures. In addition, we elicit token-level concreteness ratings from models and evaluate alignment to human norm distributions, testing whether multimodal training yields more human-consistent judgments. Across benchmarks and scales, VLMs show larger gains on more concrete inputs, exhibit clearer concreteness-structured representations, produce ratings that better match human norms, and display systematically different attention patterns consistent with increased grounding.

Figures (5)

• Figure 1: Comparison of concreteness rating distributions for selected words. For each word, we plot the empirical distribution of model-generated token ratings from Llama Vision (VLM) and Llama Text (LLM), alongside human norms from brysbaert2014.
• Figure 2: Top row: accuracy by question concreteness for Llama Text vs. Llama Vision. Bottom row: the VLM--LLM accuracy gap.
• Figure 3: t-SNE of average last-layer token representations for Llama Text vs. Llama Vision, colored by human concreteness.
• Figure 4: Layerwise Pearson's $r$ between token concreteness and head-averaged attention entropy. Colored dash lines are sigmoid fitted curves.
• Figure 5: Human--model alignment of token-level concreteness judgments. Each point is a word with human concreteness on the x-axis and symmetric KL divergence between the model’s rating distribution and the human (40K) distribution on the y-axis (lower is better). Larger dots show bin averages; dashed lines are linear fits over bins. The VLM (left) exhibits lower divergence and a steeper decrease in divergence with concreteness than the matched text-only LLM (right).