Quantifying Semantic Emergence in Language Models

TL;DR

This work introduces Information Emergence (IE), a principled, task-agnostic metric designed to quantify how effectively large language models extract semantics from token sequences. IE compares entropy reductions at macro (semantic) versus micro (token) levels across transformer blocks, estimated with a lightweight mutual-information estimator to accommodate large models. Through synthetic in-context learning (ICL) and natural-sentence experiments, the authors show IE tracks semantic determinability, scales with model size, and reveals distinct patterns for ICL versus natural text, including links to hallucination and generation-origin signals. The proposed framework offers a tractable and interpretable lens on semantic emergence in LLMs, with potential applications in model comparison, hallucination analysis, and authorship attribution among generated texts.

Abstract

Large language models (LLMs) are widely recognized for their exceptional capacity to capture semantics meaning. Yet, there remains no established metric to quantify this capability. In this work, we introduce a quantitative metric, Information Emergence (IE), designed to measure LLMs' ability to extract semantics from input tokens. We formalize ``semantics'' as the meaningful information abstracted from a sequence of tokens and quantify this by comparing the entropy reduction observed for a sequence of tokens (macro-level) and individual tokens (micro-level). To achieve this, we design a lightweight estimator to compute the mutual information at each transformer layer, which is agnostic to different tasks and language model architectures. We apply IE in both synthetic in-context learning (ICL) scenarios and natural sentence contexts. Experiments demonstrate informativeness and patterns about semantics. While some of these patterns confirm the conventional prior linguistic knowledge, the rest are relatively unexpected, which may provide new insights.

Figures (6)

• Figure 1: The analogy of auto-regressive process in NTP to Markov process. Taking the output representation of token$2$ in Block $0$ ($h^{2}_{1}$) as an example, which receives information from input representations of $h^{0}_{0}$, $h^{1}_{0}$, and $h^{2}_{0}$,satisfying $p_{h^{2}_{l+1}|H_{l}^{\leqslant t}} = p_{h^{2}_{l+1}|h^{0}_{l},h^{1}_{l},h^{2}_{l}}$.
• Figure 2: The increasement of IE, EM, Accuracy, and model loss for GPT2-XL in comparison to the previous token: $\text{increasement}=(value(t)-value(t-1))/value(t)$, where $value(t)$ represents the value at token $t$. Therefore, a positive increasement $(>0)$ indicates an increase in the metric value, and a decrease vice versa.
• Figure 3: IE and Model Performance with model size increasing in Arithmetic
• Figure 4: $\widehat{E}(t)$ on ICL and natural scenarios with mean and variance.
• Figure 5: Mutual information of each token position in two datasets, taking GPT2-XL and GENNA as examples.

Theorems & Definitions (2)

• Example 1
• Definition 1: Information Emergence in LLMs