Think Globally, Group Locally: Evaluating LLMs Using Multi-Lingual Word Grouping Games

TL;DR

GlobalGroup introduces a multilingual abstract reasoning benchmark derived from the NYT Connections game to evaluate LLMs across English, Spanish, Chinese, Hindi, and Arabic. The framework combines native-language groupings with English translations, a ground-truth matching and group-topic scoring scheme, and difficulty analyses based on group size, embedding-based clustering, and word overlap. Key findings show a pronounced English-language bias, substantial performance gaps between open- and closed-source models, and that larger open models can rival larger closed models on non-English tasks, with three difficulty metrics enabling finer-grained analysis. This work provides a practical, scalable approach to measuring linguistic biases in abstract reasoning and highlights the importance of multilingual training for equitable model performance across languages and cultures.

Abstract

Large language models (LLMs) can exhibit biases in reasoning capabilities due to linguistic modality, performing better on tasks in one language versus another, even with similar content. Most previous works evaluate this through reasoning tasks where reliance on strategies or knowledge can ensure success, such as in commonsense or math tasks. However, abstract reasoning is vital to reasoning for everyday life, where people apply "out-of-the-box thinking" to identify and use patterns for solutions, without a reliance on formulaic approaches. Comparatively, little work has evaluated linguistic biases in this task type. In this paper, we propose a task inspired by the New York Times Connections: GlobalGroup, that evaluates models in an abstract reasoning task across several languages. We constructed a game benchmark with five linguistic backgrounds -- English, Spanish, Chinese, Hindi, and Arabic -- in both the native language and an English translation for comparison. We also proposed game difficulty measurements to evaluate models on games with similar difficulty, enabling a more controlled comparison, which is particularly important in reasoning evaluations. Through experimentation, we find English modalities largely lead to better performance in this abstract reasoning task, and performance disparities between open- and closed-source models.

Figures (7)

• Figure 1: An example Spanish (es) GlobalGroup game. The model is given 12 words and is instructed to output 3 groups of 4 words, along with the topics it identifies connects each group of words. In the figure, same-colored words form a group. The model made mistakes in its grouping.
• Figure 2: PCA plot of model representations based on their performance. Each point corresponds to a model, projected onto the first two principal components.
• Figure 3: Depicting the distribution of game average F1 and CTD scores across 5 buckets of score ranges, encapsulating the total score range for both metrics. This is evaluated for closed-source models.
• Figure 4: Depicting the distribution of total game composition of games with silhouette score values across 100 buckets of silhouette score ranges, for games where a model scored an F1 $\le$ 0.5. This is evaluated for closed-source models, with game silhouette scores calculated using both the true groupings, and guessed groupings for a given game.
• Figure 5: Depicting the distribution of total game composition of games with silhouette score values across 100 buckets of silhouette score ranges, for games where a model scored an F1 $>$ 0.5. This is evaluated for closed-source models, with game silhouette scores calculated using both the true groupings, and guessed groupings for a given game.