NEO-BENCH: Evaluating Robustness of Large Language Models with Neologisms

TL;DR

Neo-Bench addresses the problem of LLM robustness to neologisms, a facet of language drift underrepresented in prior work. It introduces a diverse neologism corpus (2,505 items from multiple sources) and a four-task benchmark (Perplexity, Cloze QA, Definition Generation, Machine Translation) to quantify how well models generalize to new word forms. The study reveals that older models struggle across tasks, automatic MT metrics often diverge from human judgments, and neologisms’ linguistic type shapes difficulty. Findings show larger, more recently trained models handle neologisms better, yet challenges persist, especially for semantic neologisms; the work also provides a framework for ongoing data collection and multilingual expansion. Overall, Neo-Bench offers a practical, time-aware benchmark with potential to guide model updates and evaluation in dynamic linguistic landscapes. $2{,}505$ neologisms and four evaluation tasks form the core of a resource designed to keep pace with rapid language evolution.

Abstract

The performance of Large Language Models (LLMs) degrades from the temporal drift between data used for model training and newer text seen during inference. One understudied avenue of language change causing data drift is the emergence of neologisms -- new word forms -- over time. We create a diverse resource of recent English neologisms by using several popular collection methods. We analyze temporal drift using neologisms by comparing sentences containing new words with near-identical sentences that replace neologisms with existing substitute words. Model performance is nearly halved in machine translation when a single neologism is introduced in a sentence. Motivated by these results, we construct a benchmark to evaluate LLMs' ability to generalize to neologisms with various natural language understanding tasks and model perplexity. Models with later knowledge cutoff dates yield lower perplexities and perform better in downstream tasks. LLMs are also affected differently based on the linguistic origins of words, indicating that neologisms are complex for static LLMs to address. We will release our benchmark and code for reproducing our experiments.

Figures (13)

• Figure 1: Neo-Bench collects neologisms from 2020-2023 for LLM evaluation. "Pig Butchering" originated as a Mandarin expression (杀猪盘).
• Figure 2: A single neologism can dramatically affect model output, as shown by human evaluation of Machine Translation models on sentences containing neologisms and the same sentences with neologisms replaced by carefully chosen words that also fit in the context. Oracle ensemble selects the best translation from all models.
• Figure 3: Percentage of good translations and mistranslations of neologism sentences over time. The dashed line represents the percentage of good translations achieved on non-neologism sentences.
• Figure 4: Example Google Trend lines measuring neologism prevalence. The dashed line estimates the date a neologism becomes popular while not yet conventional.
• Figure 5: Left: Results of the Cloze Question Answering task reported by accuracy of selecting the neologism or distractor option. Combined accuracy for selecting either answer is provided. Right: Results of the Definition Generation task reported with accuracy of correct definitions. 5-shot prompting of models is used for both tasks.