OmniScience: A Large-scale Multi-modal Dataset for Scientific Image Understanding

TL;DR

OmniScience tackles the bottleneck in scientific image understanding by building a large-scale, high-fidelity multi-modal dataset of $1.5$ million figure-caption-context triplets from open-access sources spanning more than $10$ disciplines. A dynamic model-routing recaptioning pipeline synthesizes visual features, original captions, and in-text contexts to produce dense, self-contained descriptions, boosting image–text alignment from $0.769$ to $0.956$ and increasing caption length to $360.6$ words on average. Quality control, deduplication, and LLM-based evaluation align captions with expert judgments and enable a novel caption QA proxy task, where finetuned models achieve substantial gains on MM-MT-Bench, MMMU, and MSEarth benchmarks. The work demonstrates that high-fidelity, richly grounded captions are crucial for effective cross-modal learning in scientific domains and positions OmniScience as a foundational dataset for future AI-for-science research.

Abstract

Multimodal Large Language Models demonstrate strong performance on natural image understanding, yet exhibit limited capability in interpreting scientific images, including but not limited to schematic diagrams, experimental characterizations, and analytical charts. This limitation is particularly pronounced in open-source MLLMs. The gap largely stems from existing datasets with limited domain coverage, coarse structural annotations, and weak semantic grounding. We introduce OmniScience, a large-scale, high-fidelity multi-modal dataset comprising 1.5 million figure-caption-context triplets, spanning more than 10 major scientific disciplines. To obtain image caption data with higher information density and accuracy for multi-modal large-model training, we develop a dynamic model-routing re-captioning pipeline that leverages state-of-the-art multi-modal large language models to generate dense, self-contained descriptions by jointly synthesizing visual features, original figure captions, and corresponding in-text references authored by human scientists. The pipeline is further reinforced with rigorous quality filtering and alignment with human expert judgments, ensuring both factual accuracy and semantic completeness, and boosts the image-text multi-modal similarity score from 0.769 to 0.956. We further propose a caption QA protocol as a proxy task for evaluating visual understanding. Under this setting, Qwen2.5-VL-3B model finetuned on OmniScience show substantial gains over baselines, achieving a gain of 0.378 on MM-MT-Bench and a gain of 0.140 on MMMU.

Figures (14)

• Figure 1: Disciplinary composition and visual diversity of OmniScience. The central pie chart summarizes the distribution of figures across major scientific fields, while surrounding examples highlight rich visual heterogeneity across domains. Together, they demonstrate the broad disciplinary coverage and diverse visual distributions of OmniScience.
• Figure 2: Overview of OmniScience data construction. The example is taken from Hu et al. hu2023lantern. Each scientific figure is represented as an image–caption–context triplet, where the raw caption is paired with its corresponding context paragraph(s) extracted from the source article to provide surrounding textual grounding. Fine-grained subfigure annotations specify each panel’s bounding box, sub-caption, and semantic type (e.g., “chemical reaction”, “microscopy”, etc.). Additionally, a recaption field provides a refined summary that integrates visual and textual cues to fully convey the scientific meaning of the figure.
• Figure 3: Caption length and image resolution comparison between the OmniScience and MMSCI datasets. (a) Probability density distributions of caption lengths (in words) for both datasets. (b) Two-dimensional density distribution of image resolutions (width and height in pixels). The pronounced vertical line in the MMSCI distribution is a processing artifact, as all images in its public release were constrained to a fixed width of 685 pixels.
• Figure 4: Distribution of multi-modal reranker scores for captions from OSrawcap (OmniScience raw captions) and OmniScience (re-captioned). The horizontal axis uses a non-linear scale.
• Figure 5: Distribution of multi-modal reranker scores for captions generated by Qwen2.5-VL-3B fine-tuned on different datasets, evaluated on the OmniScience validation set.