CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation

TL;DR

CANINE introduces a tokenization-free, vocabulary-free encoder that operates on Unicode characters, using hash-based embeddings and downsampling to enable efficient deep-transformer representations. It offers autoregressive character and subword pre-training losses, plus a modular design that discards vocab/tokenizers after pre-training, allowing untokenized input downstream. On TyDi QA and NER benchmarks, CANINE matches or surpasses mBERT with fewer parameters and shows strong performance on morphologically rich languages, aided by its downsampling and character-centric architecture. This work reduces preprocessing burdens while expanding multilingual applicability and robustness to orthographic variation.

Abstract

Pipelined NLP systems have largely been superseded by end-to-end neural modeling, yet nearly all commonly-used models still require an explicit tokenization step. While recent tokenization approaches based on data-derived subword lexicons are less brittle than manually engineered tokenizers, these techniques are not equally suited to all languages, and the use of any fixed vocabulary may limit a model's ability to adapt. In this paper, we present CANINE, a neural encoder that operates directly on character sequences, without explicit tokenization or vocabulary, and a pre-training strategy that operates either directly on characters or optionally uses subwords as a soft inductive bias. To use its finer-grained input effectively and efficiently, CANINE combines downsampling, which reduces the input sequence length, with a deep transformer stack, which encodes context. CANINE outperforms a comparable mBERT model by 2.8 F1 on TyDi QA, a challenging multilingual benchmark, despite having 28% fewer model parameters.

Figures (2)

• Figure 1: Canine neural architecture.
• Figure 2: Canine-C pre-training data preparation (§\ref{['sec:autoregressive']}). Character-wise predictions are made by an auto-regressive transformer layer that predicts then reveals one character at a time, in a shuffled order.