VisionFM: a Multi-Modal Multi-Task Vision Foundation Model for Generalist Ophthalmic Artificial Intelligence

TL;DR

VisionFM provides a foundation to foster multiple ophthalmic artificial intelligence applications, such as disease screening and diagnosis, disease prognosis, subclassification of disease phenotype, and systemic biomarker and disease prediction, with each application enhanced with expert-level intelligence and accuracy.

Abstract

We present VisionFM, a foundation model pre-trained with 3.4 million ophthalmic images from 560,457 individuals, covering a broad range of ophthalmic diseases, modalities, imaging devices, and demography. After pre-training, VisionFM provides a foundation to foster multiple ophthalmic artificial intelligence (AI) applications, such as disease screening and diagnosis, disease prognosis, subclassification of disease phenotype, and systemic biomarker and disease prediction, with each application enhanced with expert-level intelligence and accuracy. The generalist intelligence of VisionFM outperformed ophthalmologists with basic and intermediate levels in jointly diagnosing 12 common ophthalmic diseases. Evaluated on a new large-scale ophthalmic disease diagnosis benchmark database, as well as a new large-scale segmentation and detection benchmark database, VisionFM outperformed strong baseline deep neural networks. The ophthalmic image representations learned by VisionFM exhibited noteworthy explainability, and demonstrated strong generalizability to new ophthalmic modalities, disease spectrum, and imaging devices. As a foundation model, VisionFM has a large capacity to learn from diverse ophthalmic imaging data and disparate datasets. To be commensurate with this capacity, in addition to the real data used for pre-training, we also generated and leveraged synthetic ophthalmic imaging data. Experimental results revealed that synthetic data that passed visual Turing tests, can also enhance the representation learning capability of VisionFM, leading to substantial performance gains on downstream ophthalmic AI tasks. Beyond the ophthalmic AI applications developed, validated, and demonstrated in this work, substantial further applications can be achieved in an efficient and cost-effective manner using VisionFM as the foundation.

Figures (6)

• Figure 1: a. Previous ophthalmic AI models are specialized, built to be single-purpose and disease (e.g., diabetic retinopathy), often focusing on a single modality (e.g., fundus photographs) and solving a single clinical task and process (e.g. screening). b. VisionFM is a novel AI model built to be multi-purpose, multi-disease, multi-modal, multi-task foundation model that can simultaneously approach multiple ophthalmic clinical tasks with the ability of processing multiple ophthalmic imaging modalities. VisionFM shows strong generalization to previously untrained ophthalmic modality and imaging devices, and shows robust few-shot disease diagnostic accuracy.
• Figure 2: a. Data used for pre-training and evaluating VisionFM covers a diverse geographic locations (26 countries and regions in total). Pre-training data alone contains 3.4 million ophthalmic images from 560,457 unique individuals. b. The pre-training data of VisionFM covers eight main ophthalmic imaging modalities captured by a wide range of devices. c. Multi-modal data paired by diseases. DR: diabetic retinopathy. AMD: age-related macular degeneration. HR: hypertensive retinopathy. RVO: retinal vein occlusion. RD: retinal detachment.
• Figure 3: Ophthalmic disease diagnosis. a-h: Precision-recall curves of jointly recognizing eight common ophthalmic diseases on a large-scale benchmark that merges 23 public and 5 private datasets, covering five imaging modalities, and comparison with ResNet pre-trained on ImageNet and later fine-tuned on the same training set. i: AUC of recognizing eight ophthalmic diseases by VisionFM. j: Generalist diagnostic accuracy of VisionFM across 12 ophthalmic diseases and comparison with nine ophthalmic clinicians of different years of experience. k: Comparison with other self-supervised or unsupervised methods in recognizing AMD on the iChallenge-AMD dataset. l-n: Generalizability evaluation results. l: AUC of DR grading on a new modality (OCTA). m: AUC of DR grading on new imaging devices (ultra-wide-field fundus photography devices). n: AUC of diagnosing an underrepresented ophthalmic disease (ocular albinism) in few-shot manners. o: Average F1 score of VisionFM across 12 diseases, and comparison with the results of the two cohorts of ophthalmic clinicians.
• Figure 4: a: Illustration of systemic biomarker and disease prediction from ocular images. b demography of the subjects involved in systemic biomarker (top and bottom right) and disease prediction (bottom left). c: Confusion matrix of glaucoma forecasting results of VisionFM. d: Average accuracy of predicting each individual systemic biomarker from fundus and external eye images. e and f: Average accuracy of systemic biomarker prediction of different modalities and gender (e) as well as different age groups (f). g-l: Predicted and actual biomarker values of HGB, RBC, and UA (g, h, i are results of external eye images; and j, k, l are fundus images). m and n: results of predicting the presence of intracranial tumors from fundus images. m: AUC of VisionFM and comparison with clinicians. n : AP of VisionFM and comparison with clinicians.
• Figure 5: a: Quantitative segmentation results of VisionFM and comparison with U-Net. b: Few-shot OCT layer segmentation performance of VisionFM and comparison with U-Net. c. Qualitative examples of segmentation and landmark detection of VisionFM on different imaging modalities. d. Examples of synthetic slit-lamp and MRI images. e: Euclidean distance errors of VisionFM in detecting three landmarks in UBM images and comparison with U-Net. f: Cataract diagnostic performance of VisionFM with respect to different proportions of real and synthetic slit-lamp data, as well as orbital tumor segmentation accuracy of VisionFM with respect to different proportions of real and synthetic MRI data. g: Results of visual Turing test (a score close to 0.5 means that synthetic images are hard to be distinguished from real images) .