Investigating Bias: A Multilingual Pipeline for Generating, Solving, and Evaluating Math Problems with LLMs

TL;DR

The paper investigates linguistic bias in LLM-generated math explanations across English, German, and Arabic by building a fully automated pipeline that generates, translates, solves, and evaluates problems aligned with the German K-10 curriculum. It employs three commercial LLMs to produce step-by-step solutions in each language and uses a held-out panel of LLM judges to rank solution quality. The results show a consistent English advantage and Arabic disadvantages, revealing persistent multilingual bias in educational contexts. The work delivers a scalable evaluation framework and highlights the need for linguistically inclusive development to promote equitable AI-supported learning.

Abstract

Large Language Models (LLMs) are increasingly used for educational support, yet their response quality varies depending on the language of interaction. This paper presents an automated multilingual pipeline for generating, solving, and evaluating math problems aligned with the German K-10 curriculum. We generated 628 math exercises and translated them into English, German, and Arabic. Three commercial LLMs (GPT-4o-mini, Gemini 2.5 Flash, and Qwen-plus) were prompted to produce step-by-step solutions in each language. A held-out panel of LLM judges, including Claude 3.5 Haiku, evaluated solution quality using a comparative framework. Results show a consistent gap, with English solutions consistently rated highest, and Arabic often ranked lower. These findings highlight persistent linguistic bias and the need for more equitable multilingual AI systems in education.