Let's Play Across Cultures: A Large Multilingual, Multicultural Benchmark for Assessing Language Models' Understanding of Sports

TL;DR

CultSportQA presents the first large-scale, multilingual, multicultural benchmark for traditional sports, compiling 33,000 text- and image-based MCQs across 11 languages and 84 sports to probe LLMs, SLMs, and MLLMs. By evaluating zero-shot, few-shot, and chain-of-thought prompting, the study reveals clear performance gaps in culturally nuanced sports reasoning, with GPT-4o and InstructBLIP leading in their respective categories. The dataset emphasizes cultural authenticity through careful manual annotation, metadata tagging, and bias-mitigation, and it demonstrates the potential to drive more inclusive AI systems that honor regional sporting traditions. This work has practical implications for preserving cultural heritage, enriching sports journalism, and improving cross-cultural communication in AI applications, while inviting future expansion to more languages, sports, and modalities.

Abstract

Language Models (LMs) are primarily evaluated on globally popular sports, often overlooking regional and indigenous sporting traditions. To address this gap, we introduce \textbf{\textit{CultSportQA}}, a benchmark designed to assess LMs' understanding of traditional sports across 60 countries and 6 continents, encompassing four distinct cultural categories. The dataset features 33,000 multiple-choice questions (MCQs) across text and image modalities, each of which is categorized into three key types: history-based, rule-based, and scenario-based. To evaluate model performance, we employ zero-shot, few-shot, and chain-of-thought (CoT) prompting across a diverse set of Large Language Models (LLMs), Small Language Models (SLMs), and Multimodal Large Language Models (MLMs). By providing a comprehensive multilingual and multicultural sports benchmark, \textbf{\textit{CultSportQA}} establishes a new standard for assessing AI's ability to understand and reason about traditional sports.

Figures (6)

• Figure 1: CultSportQA is a diverse benchmark featuring 11 languages, with questions manually created and verified by native language experts. It covers three key aspects of traditional sports across two modalities, text and image, emphasizing mid to low-resource languages and sports originating from 11 countries across 3 continents. These sports, now played in 60 countries across 6 continents, are depicted with dark blue for their origins and light blue for their current reach. CultSportQA offers a wide range of question formats, including multiple-choice questions (MCQs) and both short and long visual question-answering (VQA) tasks.
• Figure 2: CultSportQA Manual Data Collection Pipeline: The data collection process involved two key stages. In the first stage, annotators gathered data sources and generated questions, drawing from their respective cultural backgrounds and languages. In the second stage, annotators reviewed and verified the questions to ensure cultural authenticity and maintain high translation quality.
• Figure 3: Distribution of image-based and text-based questions across different question types and continents.
• Figure 4: Zero-shot results of language models across different question types.
• Figure 5: Average results of language models on the CultSportQA dataset classified on the basis of languages.