Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models

TL;DR

This work tackles the rigidity and scalability challenges of subword tokenizers by proposing a tokenization-free hierarchical autoregressive Transformer that fuses a compact character-level encoder/decoder with a word-level backbone. The model is trained end-to-end without a fixed vocabulary and scales to 7B parameters, achieving competitive downstream performance and greater robustness to input perturbations, while enabling faster cross-domain adaptation through continued pretraining. Through architecture sweeps and compute-matched experiments, the authors show parity with tokenizer baselines across tasks, with notable gains on Lambada and strong robustness advantages. The approach also demonstrates practical benefits in cross-lingual adaptation, delivering faster training and improved retention of knowledge when exposed to out-of-domain languages and domains, suggesting broad implications for robust, generalizable NLP systems.

Abstract

Tokenization is a fundamental step in natural language processing, breaking text into units that computational models can process. While learned subword tokenizers have become the de-facto standard, they present challenges such as large vocabularies, limited adaptability to new domains or languages, and sensitivity to spelling errors and variations. To overcome these limitations, we investigate a hierarchical architecture for autoregressive language modelling that combines character-level and word-level processing. It employs a lightweight character-level encoder to convert character sequences into word embeddings, which are then processed by a word-level backbone model and decoded back into characters via a compact character-level decoder. This method retains the sequence compression benefits of word-level tokenization without relying on a rigid, predefined vocabulary. We demonstrate, at scales up to 7 billion parameters, that hierarchical transformers match the downstream task performance of subword-tokenizer-based models while exhibiting significantly greater robustness to input perturbations. Additionally, during continued pretraining on an out-of-domain language, our model trains almost twice as fast, achieves superior performance on the target language, and retains more of its previously learned knowledge. Hierarchical transformers pave the way for NLP systems that are more robust, flexible, and generalizable across languages and domains.

Figures (10)

• Figure 1: Schematic of the proposed hierarchical architecture. The input text is first split into words, with each word prepended by a special token [W]. These words are passed through the encoder and the activation at the position of the [W] token is selected as the word embedding $\mathbf{e}^i$. The sequence of word embeddings $\mathbf{e}^i$ is then processed by the backbone to produce abstract predictive word embeddings$\mathbf{p}^i$. The decoder then maps $\mathbf{p}^i$ to probabilites for the characters of the next word. During inference, given a text with a partial word, complete words are processed by the encoder and backbone. The decoder then recursively completes the remaining characters of the incomplete word and the completed word enters the encoder in a word recursion.
• Figure 2: Bytes per word (BPW) and bytes per token (BPT) statistics, showing that words are a coarser unit than subword tokens. The tokenizer has been fitted to the DCLM-baseline dataset (left). On a dataset to which the tokenizer is not attuned, such as the German Occiglot dataset (right), the tokenizer "fragments" and BPT drops significantly.
• Figure 3: Word and byte accuracy for hierarchical models with different encoder/decoder sizes. Each candidate encoder/decoder size has been trained with backbone sizes ranging from $L_\text{b}=H_\text{b}=16$ to $L_\text{b}=H_\text{b}=30$. The horizontal axis shows training compute (Eq. \ref{['eq:approximate-cost-hierarchical']}) normalized by the highest value. The vertical gray lines indicate the compute required by the baseline models. Small character-level modules tend to yield better word accuracy at the tested compute budgets.
• Figure 4: Average change in accuracy across the MMLU, OpenBookQA, Arc Challenge and HellaSwag eval tasks for perturbations to the prompt.
• Figure 5: Continued pretraining on Occiglot German. We show average eval scores on English and German. The hierarchical architecture adapts better to the new language, while also retaining higher scores on English evals. At an equal FLOP budget we are only able to train the baseline for half as many steps due to it requiring almost two times the compute per document on the german dataset. Nevertheless, we continued training the baseline for more steps, depicted as dotted lines.