From Tokens to Words: On the Inner Lexicon of LLMs

TL;DR

This work reveals that LLMs maintain an internal, word-level lexicon beyond the tokenizer and perform detokenization primarily in early-to-mid layers by aggregating subword tokens and using FFN-driven updates to reconstruct full word representations. By applying probing, logit-lens, and Patchscopes analyses, the authors show robust word vs. nonword discrimination and demonstrate word retrieval for both artificially split single-token words and multi-token out-of-vocabulary words, implying an expansive latent vocabulary. They propose a practical three-step, finetuning-free method to expand an LLM’s vocabulary by deriving detokenized representations and learning lightweight linear mappings to embed and unembed new words, achieving token-length reductions with minimal impact on accuracy across multiple domains. These findings offer a concrete pathway to improve efficiency and multilingual capability without changing base model parameters, with potential implications for token management and domain adaptation.

Abstract

Natural language is composed of words, but modern large language models (LLMs) process sub-words as input. A natural question raised by this discrepancy is whether LLMs encode words internally, and if so how. We present evidence that LLMs engage in an intrinsic detokenization process, where sub-word sequences are combined into coherent whole-word representations at their last token. Our experiments show that this process primarily takes place within the early and middle layers of the model. We further demonstrate its robustness to arbitrary splits (e.g., "cats" to "ca" and "ts"), typos, and importantly-to out-of-vocabulary words: when feeding the last token internal representations of such words to the model as input, it can "understand" them as the complete word despite never seeing such representations as input during training. Our findings suggest that LLMs maintain a latent vocabulary beyond the tokenizer's scope. These insights provide a practical, finetuning-free application for expanding the vocabulary of pre-trained models. By enabling the addition of new vocabulary words, we reduce input length and inference iterations, which reduces both space and model latency, with little to no loss in model accuracy.

Figures (13)

• Figure 1: Left: The sub-word detokenization process in LLMs. From bottom to top: (a) Tokenization and Embedding: The input string is tokenized using a sub-word tokenizer (e.g., BPE) and converted into vector embeddings; (b) Token Aggregation: The attention mechanism relays information from the word's previous tokens ("un", "h") into its final sub-word representation ("appiness"); (c) Word Retrieval: The model retrieves the full word representation from an implicit internal lexicon in its feedforward (FFN) layers. This representation is added to the residual stream, until it takes over the word's hidden representation. Right: When taking this hidden representation and patching it into another prompt, the model interprets it as the original word. In this example, the model is prompted to repeat the (single vector) hidden representation of the word "unhappiness" (originally represented as three tokens), and is able to "understand" it by regenerating the original three tokens.
• Figure 2: Our word vs. nonword probing classification experiments. (\ref{['fig:dataset_creation']}) Dataset creation process. Top: words from the Gutenberg corpus are tokenized using the Llama2 tokenizer. Bottom: nonwords are generated by shuffling tokens while maintaining their positions within the word; (\ref{['fig:validation_accuracy_nonwords']}) Classification results of words vs. nonwords. Using the last token shows strong results (reaching up to 90% accuracy), which sharply rise in early layers (3--7), peak in the middle (13), and decrease in later layers (20--32). Using the penultimate token shows substantially lower scores, suggesting that the high classification accuracy is specifically tied to the presence of a complete word rather than token co-occurrence patterns.
• Figure 3: (\ref{['fig:rate_single_tokens']}) Logit lens rate of retrieval of single-token words artificially split (blue line) and split due to typos (orange). In both cases, we see an increasing rate of retrieval after the 4th layer, peaking in the middle (16--17 layers) and then dropping; (\ref{['fig:rate_multi_tokens']}) Patchscopes rate of retrieval for multi-token words. In this case results peak in the 5th layer and then start to decline.
• Figure 4: Word retrieval in FFN layers vs. in the hidden states. (4a) Retrieval rates across layers. The FFN values peak before word retrieval begins in the hidden layers. (4b) Its cumulative version, showing word retrieval occurs earlier and more frequently in FFNs than in the hidden representation.
• Figure 5: Attention weights for 2-token words: Early peaks (layers 2--3) show high attention values from the second sub-word token to the first, but these values decline rapidly. Attention from single-token words to their previous token shows a similar trend, though with substantially lower values at first, which become higher later.