Complementary Human-AI Clinical Reasoning in Ophthalmology

TL;DR

This study evaluates AMIE, a domain-general, Gemini-based conversational AI with integrated web search and self-critique, for ophthalmology diagnostic reasoning using 100 real-world case vignettes. It shows AMIE achieves high standalone diagnostic accuracy comparable to clinicians and, crucially, enhances clinician performance after review by shifting diagnostic rankings and enriching management plans. Raters also preferred AMIE outputs in masked comparisons and demonstrated substantial plan revisions across cases, indicating a complementary AI-human interaction rather than replacement. The findings support targeted domain adaptation and prospective multimodal studies (including ocular imaging) to translate this approach into real-world, equitable eye-care improvements, while acknowledging limitations such as vignette-based fidelity and small, non-representative samples.

Abstract

Vision impairment and blindness are a major global health challenge where gaps in the ophthalmology workforce limit access to specialist care. We evaluate AMIE, a medically fine-tuned conversational system based on Gemini with integrated web search and self-critique reasoning, using real-world clinical vignettes that reflect scenarios a general ophthalmologist would be expected to manage. We conducted two complementary evaluations: (1) a human-AI interactive diagnostic reasoning study in which ophthalmologists recorded initial differentials and plans, then reviewed AMIE's structured output and revised their answers; and (2) a masked preference and quality study comparing AMIE's narrative outputs with case author reference answers using a predefined rubric. AMIE showed standalone diagnostic performance comparable to clinicians at baseline. Crucially, after reviewing AMIE's responses, ophthalmologists tended to rank the correct diagnosis higher, reached greater agreement with one another, and enriched their investigation and management plans. Improvements were observed even when AMIE's top choice differed from or underperformed the clinician baseline, consistent with a complementary effect in which structured reasoning support helps clinicians re-rank rather than simply accept the model output. Preferences varied by clinical grade, suggesting opportunities to personalise responses by experience. Without ophthalmology-specific fine-tuning, AMIE matched clinician baseline and augmented clinical reasoning at the point of need, motivating multi-axis evaluation, domain adaptation, and prospective multimodal studies in real-world settings.

Figures (21)

• Figure 1: Study overview. (A) Content creation by five ophthalmologist case authors. (B) AMIE responses used in the study. (C) Task 1---Human–AI interactive diagnostic reasoning: raters record baseline reasoning, then revise after viewing the AMIE structured response. (D) Task 2---Rater evaluation: masked comparison of the AMIE narrative response versus case author reference; rater preference and quality scoring; post-study feedback.
• Figure 2: AMIE self-critique reasoning loop for inference. AMIE first generates an initial case response, then formulates search queries to retrieve additional context. Using the retrieved information, AMIE critiques its draft and produces a revised response.
• Figure 3: Accuracy comparison: AMIE versus raters (before/after seeing AMIE's response.) Bar plot showing the Top-1, Top-2, and Top-3 diagnosis accuracy for AMIE, raters before and after seeing AMIE's response.
• Figure 4: Top-1 accuracy by subspecialty (Rater Before AMIE, Rater After AMIE, AMIE) Radar plot showing Top-1 diagnostic accuracy within each subspecialty for AMIE, raters before seeing AMIE, and the same raters after review. Radial axis: 0-100%.
• Figure 5: Percentage of revision after seeing AMIE's responses. Horizontal stacked bar plot showing the percentage of raters who revised their Differential Diagnosis (DDx) list and Management Plan after reviewing AMIE's responses.