Arabic Dataset for LLM Safeguard Evaluation

TL;DR

This work addresses the gap in Arabic LLM safety evaluation by presenting a culturally localized dataset of 5,799 questions derived from a Chinese Do-Not-Answer framework and a novel dual-perspective evaluation (governmental and opposition viewpoints). It translates and adapts content to reflect Arab-region sensitivities, introducing region-specific harm types and six risk areas, including a region-wide Risk VI. The authors collect 28,955 responses from five Arabic-centric/multilingual LLMs and perform both automatic (GPT-4o) and human evaluations, revealing higher overall vulnerability in Arabic prompts and offering insights into model-specific safety weaknesses (e.g., Llama3) and the benefits of dual-perspective analysis. The study demonstrates the necessity of regionally tailored safeguards and provides a methodological blueprint for culturally aware safety assessment with implications for safer deployment of Arabic LLMs.

Abstract

The growing use of large language models (LLMs) has raised concerns regarding their safety. While many studies have focused on English, the safety of LLMs in Arabic, with its linguistic and cultural complexities, remains under-explored. Here, we aim to bridge this gap. In particular, we present an Arab-region-specific safety evaluation dataset consisting of 5,799 questions, including direct attacks, indirect attacks, and harmless requests with sensitive words, adapted to reflect the socio-cultural context of the Arab world. To uncover the impact of different stances in handling sensitive and controversial topics, we propose a dual-perspective evaluation framework. It assesses the LLM responses from both governmental and opposition viewpoints. Experiments over five leading Arabic-centric and multilingual LLMs reveal substantial disparities in their safety performance. This reinforces the need for culturally specific datasets to ensure the responsible deployment of LLMs.

Figures (7)

• Figure 1: Arabic word cloud focusing on Region-specific sensitivity risk area.
• Figure 2: The number of harmful responses across the five LLMs on our Arabic dataset. Models were evaluated from both government and opposition perspectives for the responses to controversial questions.
• Figure 3: Harmful response distribution across: (a) five risk areas (I--V); (b) regional-specific risk area VI from government vs. opposition perspective (c). Cul=cultural preservation & social issues, Reg=regional issues, Eco=economic policies & development, Gov=governance, Dyn=political dynamics, Rep=political repression, Hum=human rights & freedom of expression, and Hist=controversial historical events.
• Figure 4: Harmful responses over direct attacks, indirect attacks, and harmless variants.
• Figure 5: Responding pattern distribution across the whole and three types of questions over five Arabic LLMs.