Afri-MCQA: Multimodal Cultural Question Answering for African Languages

TL;DR

Afri-MCQA introduces the first large-scale multilingual multimodal cultural VQA benchmark for African languages, combining text and speech in 15 languages across 12 countries with about $7.5k$ QA pairs. The study reveals that current open-weight MLLMs struggle with African cultural knowledge, especially under speech inputs, and that English queries generally outperform native-language queries due to linguistic gaps. Through diagnostic control tasks, the authors distinguish language understanding from cultural knowledge, finding substantial cross-lingual gaps and highlighting foundational speech-processing limitations (LID/ASR) as major bottlenecks. The work emphasizes the need for speech-first approaches, culturally grounded pretraining, and cross-lingual cultural transfer, and provides Afri-MCQA as a publicly available benchmark to drive inclusive multimodal AI in African languages. Overall, Afri-MCQA offers a critical evaluation platform and a data resource to foster culturally aware, language-diverse AI systems in Africa.

Abstract

Africa is home to over one-third of the world's languages, yet remains underrepresented in AI research. We introduce Afri-MCQA, the first Multilingual Cultural Question-Answering benchmark covering 7.5k Q&A pairs across 15 African languages from 12 countries. The benchmark offers parallel English-African language Q&A pairs across text and speech modalities and was entirely created by native speakers. Benchmarking large language models (LLMs) on Afri-MCQA shows that open-weight models perform poorly across evaluated cultures, with near-zero accuracy on open-ended VQA when queried in native language or speech. To evaluate linguistic competence, we include control experiments meant to assess this specific aspect separate from cultural knowledge, and we observe significant performance gaps between native languages and English for both text and speech. These findings underscore the need for speech-first approaches, culturally grounded pretraining, and cross-lingual cultural transfer. To support more inclusive multimodal AI development in African languages, we release our Afri-MCQA under academic license or CC BY-NC 4.0 on HuggingFace (https://huggingface.co/datasets/Atnafu/Afri-MCQA)

Figures (11)

• Figure 1: Examples of Afri-MCQA datapoints, containing parallel text and speech QA pairs grounded in culturally relevant images across English and native African languages.
• Figure 2: Image categories in our dataset and their distributions.
• Figure 3: Performance comparison of models on text-based question answering tasks: (a) Text MC-VQA (Multiple Choice) and (b) Text Open-ended QA in English and Native languages.
• Figure 4: Performance comparison of models on audio-based question answering tasks: (a) Audio MC-VQA (Multiple Choice) and (b) Audio Open-ended VQA in English and Native languages.
• Figure 5: Human Evaluation for Text Open-ended VQA. Accuracy across best-performing models for English and Native. We observed that, while most models perform best in the English setting, Gemini-2.5 Pro seems to perform better in the Native language.