Hierarchical Transformers Are More Efficient Language Models

TL;DR

Hourglass introduces a hierarchical Transformer architecture that shortens activations in deeper layers and then upsamples to the original resolution, dramatically reducing computation for long sequences. By carefully preserving autoregressive causality and employing varied upsampling and pooling strategies, it achieves lower perplexity and competitive or state-of-the-art results on both language modeling (enwik8) and autoregressive image generation (ImageNet32/64). The approach yields tangible efficiency gains with minimal accuracy loss, and its compatibility with multiple attention types broadens its applicability. Extensive ablations across text and vision tasks validate design choices, and the work outlines practical guidelines for building scalable, hierarchical transformers.

Abstract

Transformer models yield impressive results on many NLP and sequence modeling tasks. Remarkably, Transformers can handle long sequences which allows them to produce long coherent outputs: full paragraphs produced by GPT-3 or well-structured images produced by DALL-E. These large language models are impressive but also very inefficient and costly, which limits their applications and accessibility. We postulate that having an explicit hierarchical architecture is the key to Transformers that efficiently handle long sequences. To verify this claim, we first study different ways to downsample and upsample activations in Transformers so as to make them hierarchical. We use the best performing upsampling and downsampling layers to create Hourglass - a hierarchical Transformer language model. Hourglass improves upon the Transformer baseline given the same amount of computation and can yield the same results as Transformers more efficiently. In particular, Hourglass sets new state-of-the-art for Transformer models on the ImageNet32 generation task and improves language modeling efficiency on the widely studied enwik8 benchmark.

Figures (8)

• Figure 1: Bits-per-character vs. training cost for baseline (orange) and hierarchical Transformers (green). We observe significant perplexity improvements on enwik8 over the vanilla Transformer–XL baseline, see text for details.
• Figure 2: Hourglass - a high-level architecture overview. The arrows denote residual connections.
• Figure 3: HourglassLM
• Figure 4: An overview of our shortening approach. Different colors denote token positions. Initially, we shift right by one, which is a standard step in TransformerLM. Then, just before performing shortening, we additionally shift the tokens right by $\emph{shorten factor}-1$ to preserve the autoregressive property of the model.
• Figure 5: An example of information leak. If the shift right factor is too small, after upsampling the knowledge from the next tokens leaks to previous ones violating autoregressiveness and making decoding impossible.