Inconsistent Tokenizations Cause Language Models to be Perplexed by Japanese Grammar

TL;DR

The paper investigates how inconsistent tokenizations hamper language models' ability to obey rare Japanese grammar, focusing on the first-person psych predicate restriction. Using minimal-pair templates, it evaluates several open-source LLMs in the 7–10B range via perplexity-based metrics, revealing tokenization as a major source of misalignment. Weblab shows lower grammatical perplexities, but this advantage is attributed to uniformly poor tokenization, while Llama 3's performance improves dramatically when test data are restricted to well-tokenized sentences, achieving up to a 28x reduction in perplexity for certain constructions. Machine translation experiments corroborate the tokenizer effect, with Weblab producing grammatical evidentials more reliably than Llama 3, indicating tokenizer design and pretraining data choices are critical for enabling adherence to nuanced Japanese grammar.

Abstract

Typical methods for evaluating the performance of language models evaluate their ability to answer questions accurately. These evaluation metrics are acceptable for determining the extent to which language models can understand and reason about text in a general sense, but fail to capture nuanced capabilities, such as the ability of language models to recognize and obey rare grammar points, particularly in languages other than English. We measure the perplexity of language models when confronted with the "first person psych predicate restriction" grammar point in Japanese. Weblab is the only tested open source model in the 7-10B parameter range which consistently assigns higher perplexity to ungrammatical psych predicate sentences than grammatical ones. We give evidence that Weblab's uniformly bad tokenization is a possible root cause for its good performance, and show that Llama 3's perplexity on grammatical psych predicate sentences can be reduced by orders of magnitude (28x difference) by restricting test sentences to those with uniformly well-behaved tokenizations. We show in further experiments on machine translation tasks that language models will use alternative grammar patterns in order to produce grammatical sentences when tokenization issues prevent the most natural sentence from being output.

Figures (2)

• Figure 1: State of the art language models frequently fail to respect nuanced aspects of Japanese grammar, such as the first person psych predicate restriction, where here GPT-4o produces a sentence which is functionally identical to the ungrammatical Example \ref{['ex:ungrammaticalpsych']} in Section \ref{['sec:psychrestriction']}.
• Figure 2: Python code to generate example sentences.