Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AFRICAPTION: Establishing a New Paradigm for Image Captioning in African Languages

Mardiyyah Oduwole, Prince Mireku, Fatimo Adebanjo, Oluwatosin Olajide, Mahi Aminu Aliyu, Jekaterina Novikova

TL;DR

This work addresses the underrepresentation of African languages in multimodal AI by introducing AfriCaption, a framework comprising an 8k image-caption dataset across 20 African languages (plus English) and a vision-to-text model that combines SigLIP’s multilingual vision encoder with the NLLB decoder. It also presents a context-preserving translation pipeline that ensembles multiple MT models and uses cosine-similarity-based selection to maintain semantic fidelity, complemented by automated and human quality assessments. Empirical results show AfriCaption achieving high-quality outputs in some languages and outperforming a strong multilingual baseline in overlapping languages, though performance varies across languages due to data balance. The work lays a foundation for inclusive, scalable multimodal AI for Africa and encourages community-driven validation and refinement to expand coverage and cultural relevance.

Abstract

Multimodal AI research has overwhelmingly focused on high-resource languages, hindering the democratization of advancements in the field. To address this, we present AfriCaption, a comprehensive framework for multilingual image captioning in 20 African languages and our contributions are threefold: (i) a curated dataset built on Flickr8k, featuring semantically aligned captions generated via a context-aware selection and translation process; (ii) a dynamic, context-preserving pipeline that ensures ongoing quality through model ensembling and adaptive substitution; and (iii) the AfriCaption model, a 0.5B parameter vision-to-text architecture that integrates SigLIP and NLLB200 for caption generation across under-represented languages. This unified framework ensures ongoing data quality and establishes the first scalable image-captioning resource for under-represented African languages, laying the groundwork for truly inclusive multimodal AI.

AFRICAPTION: Establishing a New Paradigm for Image Captioning in African Languages

TL;DR

This work addresses the underrepresentation of African languages in multimodal AI by introducing AfriCaption, a framework comprising an 8k image-caption dataset across 20 African languages (plus English) and a vision-to-text model that combines SigLIP’s multilingual vision encoder with the NLLB decoder. It also presents a context-preserving translation pipeline that ensembles multiple MT models and uses cosine-similarity-based selection to maintain semantic fidelity, complemented by automated and human quality assessments. Empirical results show AfriCaption achieving high-quality outputs in some languages and outperforming a strong multilingual baseline in overlapping languages, though performance varies across languages due to data balance. The work lays a foundation for inclusive, scalable multimodal AI for Africa and encourages community-driven validation and refinement to expand coverage and cultural relevance.

Abstract

Multimodal AI research has overwhelmingly focused on high-resource languages, hindering the democratization of advancements in the field. To address this, we present AfriCaption, a comprehensive framework for multilingual image captioning in 20 African languages and our contributions are threefold: (i) a curated dataset built on Flickr8k, featuring semantically aligned captions generated via a context-aware selection and translation process; (ii) a dynamic, context-preserving pipeline that ensures ongoing quality through model ensembling and adaptive substitution; and (iii) the AfriCaption model, a 0.5B parameter vision-to-text architecture that integrates SigLIP and NLLB200 for caption generation across under-represented languages. This unified framework ensures ongoing data quality and establishes the first scalable image-captioning resource for under-represented African languages, laying the groundwork for truly inclusive multimodal AI.
Paper Structure (19 sections, 1 equation, 5 figures, 7 tables)

This paper contains 19 sections, 1 equation, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Harms of Misrepresentation
  5. Context-Preserving Pipeline
  6. AfriCaption Dataset
  7. Data Selection and Translation
  8. Quality Assurance
  9. Results
  10. AfriCaption Model
  11. Encoder
  12. Decoder
  13. Vision-Encoder–Text-Decoder Integration
  14. Training
  15. Results and Analysis
  16. ...and 4 more sections

Figures (5)

  • Figure 1: The context-preserving adaptive pipeline, ensuring continuous improvement and high data quality.
  • Figure 2: AfriCaption model architecture.
  • Figure 3: A plot of the average human score per language with error bars denoting standard deviation.
  • Figure 4: Score distributions for Yoruba, Igbo, and Hausa. We observe that over 50% of Yoruba ratings were 8 or above, and $\sim$20% were perfect 10s. Hausa’s distribution is flatter, with a mode around 10 (16% of scores were 10) but also a substantial portion of low scores (1–4 ratings made up $\sim$35% of Hausa responses, compared to only $\sim$18% for Yoruba). Igbo’s distribution is bimodal – it has a high incidence of 9–10 scores (about one-third of Igbo ratings were 9 or 10, similar to Yoruba) and a noticeable chunk of very low scores (1’s, 2’s, 3’s accounted for $\sim$21% in Igbo, versus $\sim$9% in Yoruba). This bimodality aligns with the earlier observation of Igbo translations being hit-or-miss
  • Figure 5: Average word count per language in the AFRICaption dataset. The plot highlights variations in caption lengths across different languages.