SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabia

TL;DR

SaudiCulture tackles the gap in LLM cultural competence by introducing a Saudi Arabia–focused benchmark that captures regional diversity across five geographic areas and multiple cultural domains. The dataset contains 441 culturally rich questions, distributed across general and region-specific content, and implemented in open-ended, single-choice, and multi-answer formats to probe both generative and discriminative capabilities. Evaluations of five LLMs, including GPT-4 and Arabic-centric models, reveal systematic regional and format-dependent performance declines, with open-ended and multi-answer queries proving the most challenging. The work demonstrates the need for region-aware training data and knowledge representations, and positions SaudiCulture as a foundation for future research aimed at building more culturally aware and globally equitable AI systems.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language processing; however, they often struggle to accurately capture and reflect cultural nuances. This research addresses this challenge by focusing on Saudi Arabia, a country characterized by diverse dialects and rich cultural traditions. We introduce SaudiCulture, a novel benchmark designed to evaluate the cultural competence of LLMs within the distinct geographical and cultural contexts of Saudi Arabia. SaudiCulture is a comprehensive dataset of questions covering five major geographical regions, such as West, East, South, North, and Center, along with general questions applicable across all regions. The dataset encompasses a broad spectrum of cultural domains, including food, clothing, entertainment, celebrations, and crafts. To ensure a rigorous evaluation, SaudiCulture includes questions of varying complexity, such as open-ended, single-choice, and multiple-choice formats, with some requiring multiple correct answers. Additionally, the dataset distinguishes between common cultural knowledge and specialized regional aspects. We conduct extensive evaluations on five LLMs, such as GPT-4, Llama 3.3, FANAR, Jais, and AceGPT, analyzing their performance across different question types and cultural contexts. Our findings reveal that all models experience significant performance declines when faced with highly specialized or region-specific questions, particularly those requiring multiple correct responses. Additionally, certain cultural categories are more easily identifiable than others, further highlighting inconsistencies in LLMs cultural understanding. These results emphasize the importance of incorporating region-specific knowledge into LLMs training to enhance their cultural competence.

Figures (6)

• Figure 1: Regional cultural diversity in Saudi Arabia, showcasing traditional food, crafts, and clothing across different areas. This highlights the rich cultural landscape that LLMs must understand to provide contextually relevant responses
• Figure 2: Overview of the data collection methodology for SaudiCulture. The process starts with gathering information from diverse sources and organizing the questions by regions and cultural categories. Questions are then assigned one of three formats. The result is the SaudiCulture dataset, comprising 441 culturally rich questions.
• Figure 3: Showing an illustration of the prompt used in this work.
• Figure 4: Model accuracy across the five regions and general cultural questions. Each region is represented with a distinct color, and the bars display the performance of the five models: GPT-4, Llama, FANAR, Jais, and AceGPT. This visualization provides an overview of how well each model performs in handling culturally specific and general queries.
• Figure 5: Performance comparison of the models across different question formats for multiple regions in Saudi Arabia. The figure includes: (1) Common questions, which involve the most common food, clothing, craft, and other cultural elements, evaluated through both Open-Ended and Single-Answer formats. (2) Specialized cultural questions applicable to a specific region but not to other regions in Saudi Arabia, evaluated through Open-Ended and Single-Answer formats. (3) Specialized cultural questions with more than one correct answer, evaluated through the Multi-Answer format. Each subplot compares the accuracy scores of LLMs across the central, eastern, northern, southern, and western regions.