LongTail-Swap: benchmarking language models' abilities on rare words

TL;DR

LT-Swap addresses a blind spot in LM evaluation by measuring rare-word generalization in the long tail, using a dataset-dependent, zero-shot framework built from BabyLM corpora. It combines three subtasks—WordSwap (semantic), InflectionSwap (syntactic inflection), and AgreementSwap (syntactic agreement)—into quadruplets generated and filtered with a large language model, enabling sentence-pair discrimination without finetuning. The results show a pronounced frequency effect, with greater cross-architecture variation in the long tail, and demonstrate that more pretraining data improves rare-word understanding; a simple RAG-like prefix further boosts semantic scores, indicating in-context learning capabilities. The work provides a public framework for generating LT-Swap benchmarks and highlights the need to evaluate long-tail word learning in data-efficient LM research.

Abstract

Children learn to speak with a low amount of data and can be taught new words on a few-shot basis, making them particularly data-efficient learners. The BabyLM challenge aims at exploring language model (LM) training in the low-data regime but uses metrics that concentrate on the head of the word distribution. Here, we introduce LongTail-Swap (LT-Swap), a benchmark that focuses on the tail of the distribution, i.e., measures the ability of LMs to learn new words with very little exposure, like infants do. LT-Swap is a pretraining corpus-specific test set of acceptable versus unacceptable sentence pairs that isolate semantic and syntactic usage of rare words. Models are evaluated in a zero-shot fashion by computing the average log probabilities over the two members of each pair. We built two such test sets associated with the 10M words and 100M words BabyLM training sets, respectively, and evaluated 16 models from the BabyLM leaderboard. Our results not only highlight the poor performance of language models on rare words but also reveal that performance differences across LM architectures are much more pronounced in the long tail than in the head. This offers new insights into which architectures are better at handling rare word generalization. We've also made the code publicly avail

Figures (7)

• Figure 1: Overview of our framework to create the dataset-dependant LT-Swap tasks. The candidate LM is evaluated on sentences, generated by LLM, that contain long-tail words from its pretraining set. S1* and S2* are obtained by swapping the long-tail words from S1 and S2, thereby creating incorrect sentences. A last LLM filtering step assert that S1* and S2* are indeed less likely than S1 and S2.
• Figure 2: BLiMP and LT-Swap scores across LMs pretrained on BabyLM10M or BabyLM100M. Both LT-Swap and BLiMP are accuracy scores with 50% random chance.
• Figure 3: LT-Swap scores (i.e average of WordSwap, InflSwap and AgrSwap) broken down across frequency bins for selected architetures pretrained on BabyLM10M and BabyLM100M (16 models in total). The star symbols is a reminder that the bin 0 is only the average of AgreementSwap and InflectionSwap which explains why it is shifted up compare to the bin 1. The standard error for the Swap scores is less than 0.02 while for BLiMP the standard error than go as high as 0.2.
• Figure 4: LT-Swap scores averaged across LMs. The standard error on those scores is less than 0.02. The star symbols is a reminder that the bin 0 is the average of only AgreementSwap and InflectionSwap, not WordSwap which does not apply on this bin.
• Figure 5: AgreementSwap short and long distance scores averaged over LMs for 10M and 100M pretraining size. The standard error on those scores is less than 0.03.