The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities

TL;DR

The paper investigates whether language models learn a modality-agnostic semantic hub that places semantically equivalent inputs from different languages and modalities near each other in intermediate representations. Using a logit-lens interpretability probe and cross-type similarity tests, the authors show consistent evidence of a shared hub across multilingual text, arithmetic, code, formal semantics, vision, and audio, with English often anchoring the space in dominant-language models. They further demonstrate causal influence by intervening in the hub with English continuations, yielding predictable changes in outputs across other data types, suggesting the hub is functionally utilized rather than a byproduct of training. The work challenges the view of strictly isolated, type-specific subspaces and offers a framework for interpreting and potentially controlling multimodal models through a unified semantic space, while also noting biases and limitations tied to dominant-language data.

Abstract

Modern language models can process inputs across diverse languages and modalities. We hypothesize that models acquire this capability through learning a shared representation space across heterogeneous data types (e.g., different languages and modalities), which places semantically similar inputs near one another, even if they are from different modalities/languages. We term this the semantic hub hypothesis, following the hub-and-spoke model from neuroscience (Patterson et al., 2007) which posits that semantic knowledge in the human brain is organized through a transmodal semantic "hub" which integrates information from various modality-specific "spokes" regions. We first show that model representations for semantically equivalent inputs in different languages are similar in the intermediate layers, and that this space can be interpreted using the model's dominant pretraining language via the logit lens. This tendency extends to other data types, including arithmetic expressions, code, and visual/audio inputs. Interventions in the shared representation space in one data type also predictably affect model outputs in other data types, suggesting that this shared representations space is not simply a vestigial byproduct of large-scale training on broad data, but something that is actively utilized by the model during input processing.

Figures (14)

• Figure 1: Examples of the semantic hub effect across input data types. For every other layer, we show the closest output token to the hidden state based on the logit lens. Llama-3's hidden states are often closest to English tokens when processing Chinese texts, arithmetic expressions, and code, in a semantically corresponding way. LLaVA, a vision-language model, and SALMONN, an audio-language model, have similar behavior when processing images/audio. As shown for the arithmetic expression example, models can be intervened cross-lingually or cross-modally, such as using English even though the input is non-English, and be steered towards corresponding effects. Boldface is only for emphasis.
• Figure 2: An illustration of our hypothesis, where semantically equivalent inputs (across data types) have similar representations, and this representation is close to the continuation token in the dominant data type. Here, $z^\star$ is English and $z^\circ$ is Chinese.
• Figure 3: Results for the multilingual experiments. The 95% CI is plotted in all. Parallel texts have similar representations. Hidden states for Chinese texts are close to the unembedding of English tokens.
• Figure 4: Language probabilities of English and Chinese (and the top language, when it is neither, which only happens for Bloom). Regardless of the input language, the dominant LM language is more salient in the early-middle layers, and the input language is more salient in the final layers. Bloom does not have a clear intermediate latent language.
• Figure 5: Results for the arithmetic experiments. The 95% CI is plotted in all. Expressions in Arabic numerals have similar representation as corresponding expressions in English, as well as the unembeddings of corresponding number words in English.