Real-World Deployment and Evaluation of Kwame for Science, An AI Teaching Assistant for Science Education in West Africa

TL;DR

The paper tackles the gap in scalable science education in Africa by extending Kwame to Kwame for Science, a retrieval-based AI teaching assistant that delivers three passages as answers and five related past-WASSCE questions. It builds a domain-specific open corpus (527K passages plus 3.5K expert-answered Q&As from 1993–2021), uses SBERT for fast retrieval, and employs a topic-detection SVM to categorize questions into 48 syllabus topics. Real-world deployment over 8 months with 750 users across 32 countries yields an 87.2% top-3 accuracy and detailed usage analytics, alongside insights into data-quality challenges such as copyright, OCR, and user engagement. The work demonstrates the feasibility and impact of localized, scalable AI-assisted education in West Africa and provides a blueprint for future improvements, including real-time topic classification, RAG extensions, and broader accessibility.

Abstract

Africa has a high student-to-teacher ratio which limits students' access to teachers for learning support such as educational question answering. In this work, we extended Kwame, a bilingual AI teaching assistant for coding education, adapted it for science education, and deployed it as a web app. Kwame for Science provides passages from well-curated knowledge sources and related past national exam questions as answers to questions from students based on the Integrated Science subject of the West African Senior Secondary Certificate Examination (WASSCE). Furthermore, students can view past national exam questions along with their answers and filter by year, question type, and topics that were automatically categorized by a topic detection model which we developed (91% unweighted average recall). We deployed Kwame for Science in the real world over 8 months and had 750 users across 32 countries (15 in Africa) and 1.5K questions asked. Our evaluation showed an 87.2% top 3 accuracy (n=109 questions) implying that Kwame for Science has a high chance of giving at least one useful answer among the 3 displayed. We categorized the reasons the model incorrectly answered questions to provide insights for future improvements. We also share challenges and lessons with the development, deployment, and human-computer interaction component of such a tool to enable other researchers to deploy similar tools. With a first-of-its-kind tool within the African context, Kwame for Science has the potential to enable the delivery of scalable, cost-effective, and quality remote education to millions of people across Africa.

Figures (2)

• Figure 1: Screenshots of QA feature of Kwame for Science
• Figure 2: Architecture of QA feature of Kwame for Science