Do Vision and Language Models Share Concepts? A Vector Space Alignment Study

TL;DR

This study investigates whether representations learned by language models (LMs) structurally align with those learned by vision models (VMs). Using fourteen LMs across four families (BERT, GPT-2, OPT, LLaMA-2) and three VM architectures (ResNet, SegFormer, MAE), the authors build bimodal dictionaries from ImageNet-21K and EN-CLDI and apply Procrustes-based linear mapping to align VM embeddings to LM embeddings. They find that larger LMs develop geometries increasingly similar to VM spaces, enabling image-to-language retrieval with high precision (P@100 up to about 64%), while simple baselines perform near zero. Alignment is modulated by dispersion, polysemy, and frequency, with lower dispersion and less polysemy, and low-frequency aliases yielding stronger mappings; results persist across POS generalization, suggesting emergent referential semantics from text-based training. The findings have implications for multi-modal processing, the LM understanding debate, emergent properties, and philosophy, while acknowledging limitations such as reliance on supervised alignment and unsupervised mapping challenges. Overall, the work provides evidence that vision and language models share structural Representations, and that model scaling amplifies this cross-modal similarity, with potential applications in cross-lingual transfer and retrieval tasks.

Abstract

Large-scale pretrained language models (LMs) are said to ``lack the ability to connect utterances to the world'' (Bender and Koller, 2020), because they do not have ``mental models of the world' '(Mitchell and Krakauer, 2023). If so, one would expect LM representations to be unrelated to representations induced by vision models. We present an empirical evaluation across four families of LMs (BERT, GPT-2, OPT and LLaMA-2) and three vision model architectures (ResNet, SegFormer, and MAE). Our experiments show that LMs partially converge towards representations isomorphic to those of vision models, subject to dispersion, polysemy and frequency. This has important implications for both multi-modal processing and the LM understanding debate (Mitchell and Krakauer, 2023).

Figures (10)

• Figure 1: Mapping from MAE$_{\text{Huge}}$ (images) to OPT$_{\text{30B}}$ (text). Gold labels are in green.
• Figure 2: Experiments stages: During our experiments, words, sentences, and images are selected from the aliases list (wordlist and ImageNet-21K aliases), Wikipedia and ImageNet-21K, respectively. The source and target spaces are constructed utilizing image and word embeddings which are extracted by specialized vision and language models.
• Figure 3: t-SNE plot of 5 words mapped from MAE$_{\text{Huge}}$ (blue) to OPT$_{\text{30B}}$ (orange) using Procrustes analysis. The green represent the mapped MAE$_{\text{Huge}}$ embeddings.
• Figure 4: LMs converge toward the geometry of visual models as they grow larger on Exclude-1K set.
• Figure 5: Examples featuring the 100 nearest neighbors in the mapping of image classes into the language representation space (from MAE$_{\text{Huge}}$ to OPT$_{\text{30B}}$). The golden labels are highlighted in green.