Testing Cross-Lingual Text Comprehension In LLMs Using Next Sentence Prediction
Ritesh Sunil Chavan, Jack Mostow
TL;DR
The paper probes whether modern LLMs truly understand cross-lingual narratives or merely leverage English data abundance by introducing a large-scale cross-lingual Next Sentence Prediction (NSP) benchmark with 10,000 questions per language (English, Swahili, Hausa). It evaluates GPT-4 Turbo, Gemini 1.5 Flash, and LLaMA 3 70B under direct answering and Chain-of-Thought prompting, augmented with semantic similarity and perplexity features, to analyze factors guiding model decisions. Key contributions include the construction of a balanced multilingual NSP dataset, a cross-model evaluation revealing pronounced resource-level gaps, and a nuanced analysis of Chain-of-Thought effects, decision factors, and error modes. The findings show that while high-resource English enables strong performance, cross-lingual comprehension deteriorates in low-resource languages, and CoT can help weaker models but may hinder stronger ones, informing both educational deployment and future multilingual AI research.
Abstract
While large language models are trained on massive datasets, this data is heavily skewed towards English. Does their impressive performance reflect genuine ability or just this data advantage? To find out, we tested them in a setting where they could not rely on data abundance: low-resource languages. Building on prior work Agarwal et al. (2025) that used Next Sentence Prediction (NSP) as a test, we created a large-scale benchmark with 10,000 questions each for English (a high-resource language), Swahili (medium-resource), and Hausa (low-resource). We then tested several top models, including GPT-4 Turbo, Gemini 1.5 Flash, and LLaMA 3 70B, to see how their performance holds up. The results painted a clear picture of how levels of language resources impact outcomes. While all models excelled in English, their accuracy dropped in Swahili and fell sharply in Hausa, with LLaMA 3 struggling the most. The story became even more interesting when we introduced Chain-of-Thought (CoT) prompting. For the struggling LLaMA 3, CoT acted as a helpful guide, significantly boosting its accuracy. However, for the more capable GPT-4 and Gemini, the same technique often backfired, leading to a kind of "overthinking" that hurt their results in the cross-lingual context. This reveals that Chain-of-Thought is not a universal solution; its effectiveness depends heavily on the model's baseline capability and the specific context of the task. Our framework pinpoints LLM weaknesses, highlights when CoT helps or hinders cross-lingual NSP performance, and factors influencing their decisions.