Greed is All You Need: An Evaluation of Tokenizer Inference Methods

TL;DR

This paper investigates how the decoding or inference method used to map a fixed subword vocabulary back to text influences linguistic representation. It introduces an intrinsic, cross-cutting benchmark that combines morphological alignment, cognitive plausibility, and token-distribution metrics to evaluate seven inference methods across four vocabularies and three sizes. Key findings show that greedy inference is robust across vocabularies, while SaGe achieves state-of-the-art performance on morphology alignment; least-tokens improves cognitive metrics, but likelihood-based approaches can hurt Rényi efficiency. The work advocates decoupling vocabulary construction from inference to allow targeted improvements and efficient on-line tuning, with planned extensions to more languages and downstream tasks.

Abstract

While subword tokenizers such as BPE and WordPiece are typically used to build vocabularies for NLP models, the method of decoding text into a sequence of tokens from these vocabularies is often left unspecified, or ill-suited to the method in which they were constructed. We provide a controlled analysis of seven tokenizer inference methods across four different algorithms and three vocabulary sizes, performed on a novel intrinsic evaluation suite we curated for English, combining measures rooted in morphology, cognition, and information theory. We show that for the most commonly used tokenizers, greedy inference performs surprisingly well; and that SaGe, a recently-introduced contextually-informed tokenizer, outperforms all others on morphological alignment.