Disparities in Multilingual LLM-Based Healthcare Q&A
Ipek Baris Schlicht, Burcu Sayin, Zhixue Zhao, Frederik M. Labonté, Cesare Barbera, Marco Viviani, Paolo Rosso, Lucie Flek
TL;DR
This paper introduces Multilingual Wiki Health Care to study cross-language disparities in healthcare Q&A produced by LLMs. It analyzes how pretraining data bias (Wikipedia) translates into factual alignment across English, German, Turkish, Chinese, and Italian, and demonstrates that models tend to privilege English-language evidence. By incorporating non-English contextual excerpts and Retrieval-Augmented Generation, it shows practical routes to improve localization and reduce language-based disparities. The results reveal substantial variation in Wikipedia coverage and alignments, offering a pathway toward more equitable multilingual healthcare AI systems. The work also proposes a scalable benchmark and methodology for evaluating multilingual factuality and cross-lingual grounding in future AI healthcare tools.
Abstract
Equitable access to reliable health information is vital when integrating AI into healthcare. Yet, information quality varies across languages, raising concerns about the reliability and consistency of multilingual Large Language Models (LLMs). We systematically examine cross-lingual disparities in pre-training source and factuality alignment in LLM answers for multilingual healthcare Q&A across English, German, Turkish, Chinese (Mandarin), and Italian. We (i) constructed Multilingual Wiki Health Care (MultiWikiHealthCare), a multilingual dataset from Wikipedia; (ii) analyzed cross-lingual healthcare coverage; (iii) assessed LLM response alignment with these references; and (iv) conducted a case study on factual alignment through the use of contextual information and Retrieval-Augmented Generation (RAG). Our findings reveal substantial cross-lingual disparities in both Wikipedia coverage and LLM factual alignment. Across LLMs, responses align more with English Wikipedia, even when the prompts are non-English. Providing contextual excerpts from non-English Wikipedia at inference time effectively shifts factual alignment toward culturally relevant knowledge. These results highlight practical pathways for building more equitable, multilingual AI systems for healthcare.