Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter It

TL;DR

This work interrogates whether vision-language training reshapes the linguistic, taxonomic knowledge of language models. By constructing TaxonomiGQA, a text-only QA dataset derived from GQA that probes taxonomic reasoning via hypernym substitutions, the authors compare seven LM–VL model pairs and find that VL models generally outperform their text-only counterparts on taxonomic QA, even when the task is purely linguistic. Importantly, multiple analyses (TAXOMPS, RSA, contextual similarity, PCA) indicate VL training does not substantially change the underlying taxonomic knowledge but improves its deployment in task contexts requiring taxonomic reasoning. A preliminary investigation suggests visual similarity among hyponyms within a hypernym correlates with deployment success, hinting at how visual grounding can support generalization in taxonomic tasks. The results generalize to additional taxonomic benchmarks (Rodriguez dataset) and imply that VL training enhances the practical use of taxonomy rather than its encoded content, with implications for multimodal grounding and task-specific supervision.

Abstract

Does vision-and-language (VL) training change the linguistic representations of language models in meaningful ways? Most results in the literature have shown inconsistent or marginal differences, both behaviorally and representationally. In this work, we start from the hypothesis that the domain in which VL training could have a significant effect is lexical-conceptual knowledge, in particular its taxonomic organization. Through comparing minimal pairs of text-only LMs and their VL-trained counterparts, we first show that the VL models often outperform their text-only counterparts on a text-only question-answering task that requires taxonomic understanding of concepts mentioned in the questions. Using an array of targeted behavioral and representational analyses, we show that the LMs and VLMs do not differ significantly in terms of their taxonomic knowledge itself, but they differ in how they represent questions that contain concepts in a taxonomic relation vs. a non-taxonomic relation. This implies that the taxonomic knowledge itself does not change substantially through additional VL training, but VL training does improve the deployment of this knowledge in the context of a specific task, even when the presentation of the task is purely linguistic.

Figures (5)

• Figure 1: The three-step pipeline to create TaxonomiGQA.
• Figure 2: Performance of VLM-LM model pairs on TaxonomiGQA (\ref{['sec:gqa-exp']}) and TAXOMPS (\ref{['sec:taxomps']}). Points above the line indicate that VLM outperforms LM.
• Figure 3: Contextualized representational analysis on Qwen2.5-I and Qwen2.5-VL-I.
• Figure 4: Hypernym-specific random effects of image similarity in predicting VLM accuracy on TaxonomiGQA. Greater values indicate closer association of visual similarity to model accuracy. Bar colors indicate percentage of hypernym-hyponym pairs that have above-median similarity.
• Figure 5: Pairwise similarities between concepts in WordNet, and the transformed unembedding spaces in Qwen2.5-I (LM) vs. Qwen2.5-VL-I (VLM) (computed using park2024geometry's method), across all pairs.