OphthBench: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Ophthalmology

TL;DR

OphthBench presents a domain-specific benchmark to evaluate large language models in Chinese ophthalmology, organizing the clinical workflow into $5$ scenarios and $9$ tasks across $591$ questions, and evaluating $39$ LLMs with a two-prompt evaluation protocol and a CompassJudger-based OEQ assessment. The study finds that current LLMs average around $70\%$ performance, with prognosis tasks being relatively easier and education tasks harder, and shows that Chinese-origin models generally outperform non-Chinese ones. Prompt engineering substantially boosts performance (up to $175.0\%$ gains in some cases), underscoring the value of tailored prompting in specialized domains. OphthBench thus provides a practical, standardized framework to assess clinical utility, guide model development, and accelerate the responsible deployment of LLMs in Chinese ophthalmology.

Abstract

Large language models (LLMs) have shown significant promise across various medical applications, with ophthalmology being a notable area of focus. Many ophthalmic tasks have shown substantial improvement through the integration of LLMs. However, before these models can be widely adopted in clinical practice, evaluating their capabilities and identifying their limitations is crucial. To address this research gap and support the real-world application of LLMs, we introduce the OphthBench, a specialized benchmark designed to assess LLM performance within the context of Chinese ophthalmic practices. This benchmark systematically divides a typical ophthalmic clinical workflow into five key scenarios: Education, Triage, Diagnosis, Treatment, and Prognosis. For each scenario, we developed multiple tasks featuring diverse question types, resulting in a comprehensive benchmark comprising 9 tasks and 591 questions. This comprehensive framework allows for a thorough assessment of LLMs' capabilities and provides insights into their practical application in Chinese ophthalmology. Using this benchmark, we conducted extensive experiments and analyzed the results from 39 popular LLMs. Our evaluation highlights the current gap between LLM development and its practical utility in clinical settings, providing a clear direction for future advancements. By bridging this gap, we aim to unlock the potential of LLMs and advance their development in ophthalmology.

Figures (5)

• Figure 1: An illustration of our proposed benchmark. OphthBench is a specialized benchmark designed to assess LLM capabilities in Chinese ophthalmic practices. It comprises $5$ core ophthalmic scenarios, Education, Triage, Diagnosis, Treatment, and Prognosis, covering $9$ distinct tasks with single-choice, multiple-choice, and open-ended question formats. This comprehensive structure ensures a thorough evaluation of LLM performance across various key areas in ophthalmic practice.
• Figure 2: The statistics of the OphthBench dataset. With the help of three experienced Chinese ophthalmologists, OphthBench comprises $591$ questions spanning $5$ core ophthalmic scenarios and evaluates model performance across $9$ distinct tasks.
• Figure 3: The prompts utilized for evaluation. Bold fonts indicate content that needs to be filled in. English translations in parentheses are not included during the evaluation.
• Figure 4: A statistic of scoring rate of $39$ for each task. The results are collected with an advanced prompt to illustrate the practice challenge better.
• Figure 5: Impact of Prompts. We compared the responses generated by common and advanced prompts, reporting the performance enhancement across various tasks and models. Notably, the advanced prompts resulted in apparent performance improvements ($\geq 10\%$) in most cases, with the highest increase reaching up to $175.0\%$. However, since the advanced prompts utilized in this paper were not tailored to specific models or tasks, a performance decline of $62.2\%$ was also observed.