A unified multimodal understanding and generation model for cross-disciplinary scientific research

TL;DR

FuXi-Uni presents a unified multimodal framework that aligns high-dimensional scientific data with textual representations to enable cross-disciplinary understanding and generation. Built on a shared latent space with domain-specific encoders/decoders and an Earth-science extension, it delivers end-to-end capabilities for NL reasoning and numerical prediction across Earth science and biomedicine. Empirically, it achieves SOTA-like performance in 10-day global weather forecasting at $0.25^ \circ$, tropical cyclone forecast editing, and spatial downscaling, while also outperforming leading multimodal LLMs on biomedical VQA benchmarks. This cross-domain foundation can accelerate integrated scientific research by reducing architectural and data-framing gaps between disciplines, enabling scalable, instruction-driven collaboration across fields.

Abstract

Scientific discovery increasingly relies on integrating heterogeneous, high-dimensional data across disciplines nowadays. While AI models have achieved notable success across various scientific domains, they typically remain domain-specific or lack the capability of simultaneously understanding and generating multimodal scientific data, particularly for high-dimensional data. Yet, many pressing global challenges and scientific problems are inherently cross-disciplinary and require coordinated progress across multiple fields. Here, we present FuXi-Uni, a native unified multimodal model for scientific understanding and high-fidelity generation across scientific domains within a single architecture. Specifically, FuXi-Uni aligns cross-disciplinary scientific tokens within natural language tokens and employs science decoder to reconstruct scientific tokens, thereby supporting both natural language conversation and scientific numerical prediction. Empirically, we validate FuXi-Uni in Earth science and Biomedicine. In Earth system modeling, the model supports global weather forecasting, tropical cyclone (TC) forecast editing, and spatial downscaling driven by only language instructions. FuXi-Uni generates 10-day global forecasts at 0.25° resolution that outperform the SOTA physical forecasting system. It shows superior performance for both TC track and intensity prediction relative to the SOTA physical model, and generates high-resolution regional weather fields that surpass standard interpolation baselines. Regarding biomedicine, FuXi-Uni outperforms leading multimodal large language models on multiple biomedical visual question answering benchmarks. By unifying heterogeneous scientific modalities within a native shared latent space while maintaining strong domain-specific performance, FuXi-Uni provides a step forward more general-purpose, multimodal scientific models.

Figures (6)

• Figure 1: Science Unified Model (SUM) overview and applications.(a) SUM builds on a shared autoregressive Transformer backbone and general multimodal components with plug-and-play science encoders/decoders ($M{\times}$), enabling seamless extension to new scientific modalities and domains. (b) Representative tasks supported by SUM, including global weather forecasting, physically consistent enhancement of underestimated tropical-cyclone intensity, regional weather downscaling, and biomedical image understanding.
• Figure 2: FuXi-Uni outperforms ECMWF HRES in 10-day weather forecasts.a, Comparison of globally-averaged and latitude-weighted root mean square error ($\textrm{RMSE}$, first column) and anomaly correlation coefficient (ACC, second column) for weather forecasts from FuXi-Uni (blue lines) and ECMWF HRES (red lines). b, Spatial distributions of average $\textrm{RMSE}$ without latitude weighting of forecasts from FuXi-Uni (first column) and ECMWF HRES (second column) forecasts at day 10. Results are shown for three variables: 500 hPa geopotential ($\textrm{Z500}$, first row), 2-meter temperature ($\textrm{T2M}$, second row), and 10-meter wind speed ($\textrm{WS10M}$, third row), calculated using all testing data over a 1-year testing period (June 01, 2023 - June 30, 2024).
• Figure 3: FuXi-Uni improves both tropical cyclone (TC) track and intensity forecasts.a, Mean absolute error (MAE) of TC track forecasts (first column) and root mean square error (RMSE) of 10-meter wind speed ($\textrm{WS10M}$, second column) for TC intensity forecasts as a function of forecast lead times, comparing ECMWF HRES (blue lines), original FuXi-Uni (green lines), and FuXi-Uni with strengthened intensity (red lines). Evaluation is based on forecasts of 20 TCs and uses IBTrACS as the reference. b, Comparison of predicted TC intensity and structure for Typhoon Trami (2420). The figure shows the spatial distribution of $\textrm{WS10M}$ (m/s) at 12 UTC on 25 October 2024 from original FuXi-Uni (first column) and FuXi-Uni with strengthened intensity (second column), initialized at 12 UTC on 23 October 2024.
• Figure 4: FuXi-Uni outperforms bilinear interpolation in terms of downscaling ERA5 reanalysis from 1.5$^\circ$ to 0.25$^\circ$.a, Comparison of normalized differences in root mean squared error ($\textrm{RMSE}$, first row) and peak signal-to-noise ratio (PSNR, second row) of FuXi-Uni compared to bilinear interpolation in 0.25$^\circ$ resolution fields downscaled from 1.5$^\circ$ resolution ERA5. The x-axis and y-axis correspond to month of the year and UTC hour of day. Results are shown for two variables: 2-meter temperature ($\textrm{T2M}$, first row) and and 10-meter wind speed ($\textrm{WS10M}$, second row), calculated using all testing data over a 1-year testing period (June 01, 2023 - June 30, 2024). a, Comparison of example downscaled results from bilinear interpolation and FuXi-Uni at 18 UTC on February 02, 2024 for ($\textrm{T2M}$. first row) and ($\textrm{WS10M}$, second row).
• Figure 5: Representative cases sampled from VQA-RAD and PathVQA. For each image--question pair (1--5), we list the ground-truth answer and the corresponding predictions produced by LLaVA_Tri and FuXi-Uni.