Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICs

David Restrepo, Chenwei Wu, Zhengxu Tang, Zitao Shuai, Thao Nguyen Minh Phan, Jun-En Ding, Cong-Tinh Dao, Jack Gallifant, Robyn Gayle Dychiao, Jose Carlo Artiaga, André Hiroshi Bando, Carolina Pelegrini Barbosa Gracitelli, Vincenz Ferrer, Leo Anthony Celi, Danielle Bitterman, Michael G Morley, Luis Filipe Nakayama

TL;DR

This work addresses the problem of language bias in ophthalmology QA by introducing Multi-OphthaLingua, the first multilingual benchmark with paired questions across seven languages to enable direct cross-lingual evaluation. It reveals substantial performance disparities among LLMs, especially for LMIC languages, and shows that existing debiasing methods fall short. The authors propose CLARA, a cross-lingual reflective agent framework that combines translation, evaluation, knowledge augmentation, iterative retrieval, and rewriting to improve accuracy and fairness across languages. The results demonstrate improved multilingual performance and reduced bias, highlighting a path toward more equitable AI-assisted ophthalmology in resource-limited settings.

Abstract

Current ophthalmology clinical workflows are plagued by over-referrals, long waits, and complex and heterogeneous medical records. Large language models (LLMs) present a promising solution to automate various procedures such as triaging, preliminary tests like visual acuity assessment, and report summaries. However, LLMs have demonstrated significantly varied performance across different languages in natural language question-answering tasks, potentially exacerbating healthcare disparities in Low and Middle-Income Countries (LMICs). This study introduces the first multilingual ophthalmological question-answering benchmark with manually curated questions parallel across languages, allowing for direct cross-lingual comparisons. Our evaluation of 6 popular LLMs across 7 different languages reveals substantial bias across different languages, highlighting risks for clinical deployment of LLMs in LMICs. Existing debiasing methods such as Translation Chain-of-Thought or Retrieval-augmented generation (RAG) by themselves fall short of closing this performance gap, often failing to improve performance across all languages and lacking specificity for the medical domain. To address this issue, We propose CLARA (Cross-Lingual Reflective Agentic system), a novel inference time de-biasing method leveraging retrieval augmented generation and self-verification. Our approach not only improves performance across all languages but also significantly reduces the multilingual bias gap, facilitating equitable LLM application across the globe.

Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICs

TL;DR

This work addresses the problem of language bias in ophthalmology QA by introducing Multi-OphthaLingua, the first multilingual benchmark with paired questions across seven languages to enable direct cross-lingual evaluation. It reveals substantial performance disparities among LLMs, especially for LMIC languages, and shows that existing debiasing methods fall short. The authors propose CLARA, a cross-lingual reflective agent framework that combines translation, evaluation, knowledge augmentation, iterative retrieval, and rewriting to improve accuracy and fairness across languages. The results demonstrate improved multilingual performance and reduced bias, highlighting a path toward more equitable AI-assisted ophthalmology in resource-limited settings.

Abstract

Current ophthalmology clinical workflows are plagued by over-referrals, long waits, and complex and heterogeneous medical records. Large language models (LLMs) present a promising solution to automate various procedures such as triaging, preliminary tests like visual acuity assessment, and report summaries. However, LLMs have demonstrated significantly varied performance across different languages in natural language question-answering tasks, potentially exacerbating healthcare disparities in Low and Middle-Income Countries (LMICs). This study introduces the first multilingual ophthalmological question-answering benchmark with manually curated questions parallel across languages, allowing for direct cross-lingual comparisons. Our evaluation of 6 popular LLMs across 7 different languages reveals substantial bias across different languages, highlighting risks for clinical deployment of LLMs in LMICs. Existing debiasing methods such as Translation Chain-of-Thought or Retrieval-augmented generation (RAG) by themselves fall short of closing this performance gap, often failing to improve performance across all languages and lacking specificity for the medical domain. To address this issue, We propose CLARA (Cross-Lingual Reflective Agentic system), a novel inference time de-biasing method leveraging retrieval augmented generation and self-verification. Our approach not only improves performance across all languages but also significantly reduces the multilingual bias gap, facilitating equitable LLM application across the globe.

Paper Structure

This paper contains 14 sections, 2 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmark Construction, Results and Analysis
  4. Benchmark Construction
  5. Benchmark Results and LLM-Failure Analysis
  6. Debiasing LLMs for Multilingual Ophthalmology QA
  7. Baselines
  8. CLARA: Cross-LinguAl Reflective Agent
  9. Translation and Initial Evaluation Agent
  10. Knowledge Augmentation Agent
  11. Iterative Evaluation (of Retrieval) and Refinement Agent
  12. Question Rewriting Agent
  13. Debias Results Analysis and Ablation Studies.
  14. Conclusion and Broader Impacts

Figures (3)

  • Figure 1: The upper map: Global distribution of Low and Middle-Income Countries (LMICs); The lower map: Countries and regions where the languages included in our benchmark are predominantly spoken (English, Portuguese, Spanish, Filipino, Mandarin, French and Hindi). Our dataset covers 49 LMIC countries and approximately 4.5 billion population.
  • Figure 2: Left: An example question and answer in English and Portuguese; Right: Composition of the benchmark dataset by categories and subtypes.
  • Figure 3: Illustration of CLARA workflow