Tiny Transformers Excel at Sentence Compression

TL;DR

It is demonstrated that 1--3-layer transformers are capable of encoding and subsequently decoding standard English sentences into as little as a single 3-kilobyte token, implying that even small networks can learn to construct valid English sentences.

Abstract

It is staggering that words of the English language, which are on average represented by 5--6 bytes of ASCII, require as much as 24 kilobytes when served to large language models. We show that there is room for more information in every token embedding. We demonstrate that 1--3-layer transformers are capable of encoding and subsequently decoding standard English sentences into as little as a single 3-kilobyte token. Our work implies that even small networks can learn to construct valid English sentences and suggests the possibility of optimising large language models by moving from sub-word token embeddings towards larger fragments of text.

Figures (8)

• Figure 1: The distribution of sentence lengths in characters in the combined corpus. The horizontal axis shows the sentence length in characters, the vertical axis shows the number of sentences in the resulting corpus having that length.
• Figure 2: The distribution of sentence lengths in NLTK words in the combined corpus. The horizontal axis shows the sentence length in words after tokenising with NLTK, the vertical axis shows the number of sentences in the resulting corpus having that length.
• Figure 3: The distribution of sentence lengths in uncased BERT tokens in the combined corpus. The horizontal axis shows the sentence length in words after tokenising with the BERT tokeniser, the vertical axis shows the number of sentences in the resulting corpus having that length.
• Figure 4: A diagram of the model described in \ref{['section:model']}.
• Figure 5: The reconstruction accuracy of a model with $\ell=1,m=1,d=768$ on the test set plotted against the token length of test sentences. The horizontal axis shows the sentence length in tokens, the vertical axis shows the mean reconstruction accuracy for that length.