UbuntuGuard: A Culturally-Grounded Policy Benchmark for Equitable AI Safety in African Languages

TL;DR

UbuntuGuard addresses a critical safety gap for low-resource African languages by providing a culturally grounded, policy-based benchmark derived from expert adversarial queries. The authors build a two-stage pipeline to generate context-specific safety policies and multi-turn dialogues across 10 African languages, accompanied by automated and human quality controls. Through extensive evaluation of seven guardian models and six general-purpose models under English baseline, full localization, and cross-lingual scenarios, they reveal that English-centric benchmarks overestimate multilingual safety and that cross-lingual transfer is insufficient for robust African-language safety. The findings show a multilingual safety buffer exists with model scale, but coverage is uneven across languages and domains, underscoring the need for native African-policy–dialogue data and locally grounded evaluation resources. Overall, UbuntuGuard offers a pathway toward more equitable AI safety for low-resource languages and provides publicly available code to spur further research and development.

Abstract

Current guardian models are predominantly Western-centric and optimized for high-resource languages, leaving low-resource African languages vulnerable to evolving harms, cross-lingual safety failures, and cultural misalignment. Moreover, most guardian models rely on rigid, predefined safety categories that fail to generalize across diverse linguistic and sociocultural contexts. Robust safety, therefore, requires flexible, runtime-enforceable policies and benchmarks that reflect local norms, harm scenarios, and cultural expectations. We introduce UbuntuGuard, the first African policy-based safety benchmark built from adversarial queries authored by 155 domain experts across sensitive fields, including healthcare. From these expert-crafted queries, we derive context-specific safety policies and reference responses that capture culturally grounded risk signals, enabling policy-aligned evaluation of guardian models. We evaluate 13 models, comprising six general-purpose LLMs and seven guardian models across three distinct variants: static, dynamic, and multilingual. Our findings reveal that existing English-centric benchmarks overestimate real-world multilingual safety, cross-lingual transfer provides partial but insufficient coverage, and dynamic models, while better equipped to leverage policies at inference time, still struggle to fully localize African-language contexts. These findings highlight the urgent need for multilingual, culturally grounded safety benchmarks to enable the development of reliable and equitable guardian models for low-resource languages. Our code can be found online.\footnote{Code repository available at https://github.com/hemhemoh/UbuntuGuard.

Figures (6)

• Figure 1: UbuntuGuard construction pipeline. The pipeline has two stages: (A) data generation and (B) quality control and filtering. Given an input query, we first generate context-aware English policies using GPT-5 (1). These policies, together with the original queries, are used to generate multi-turn user–agent dialogues via NeMo Curator (2). Policies and dialogues are then translated into multiple target languages to form raw multilingual policy–dialogue (MDP) data (3). In the quality control stage, translation quality is evaluated using automatic assessment with GEMBA (4), followed by targeted human review (5).
• Figure 2: Heatmap showing the misclassification rate by domain for selected models for the fully translated Evaluation Scenario
• Figure 3: Heatmap showing the misclassification rate by domain for selected models for the Crosslingual Evaluation Scenario
• Figure 4: Percentage of false negatives across models in English-only, cross-lingual, and fully localized evaluation scenarios.
• Figure 5: Average error rate across models per domain (English-only scenario).