S1-MMAlign: A Large-Scale, Multi-Disciplinary Dataset for Scientific Figure-Text Understanding

TL;DR

This work introduces an AI-ready semantic enhancement pipeline that utilizes the Qwen-VL multimodal large model series to recaption images by synthesizing context from paper abstracts and citation contexts to address the pervasive issue of weak alignment in raw scientific captions.

Abstract

Multimodal learning has revolutionized general domain tasks, yet its application in scientific discovery is hindered by the profound semantic gap between complex scientific imagery and sparse textual descriptions. We present S1-MMAlign, a large-scale, multi-disciplinary multimodal dataset comprising over 15.5 million high-quality image-text pairs derived from 2.5 million open-access scientific papers. Spanning disciplines from physics and biology to engineering, the dataset captures diverse visual modalities including experimental setups, heatmaps, and microscopic imagery. To address the pervasive issue of weak alignment in raw scientific captions, we introduce an AI-ready semantic enhancement pipeline that utilizes the Qwen-VL multimodal large model series to recaption images by synthesizing context from paper abstracts and citation contexts. Technical validation demonstrates that this enhancement significantly improves data quality: SciBERT-based pseudo-perplexity metrics show reduced semantic ambiguity, while CLIP scores indicate an 18.21% improvement in image-text alignment. S1-MMAlign provides a foundational resource for advancing scientific reasoning and cross-modal understanding in the era of AI for Science. The dataset is publicly available at https://huggingface.co/datasets/ScienceOne-AI/S1-MMAlign.

Figures (6)

• Figure 1: Subject Distribution of S1-MMAlign. Physics (33%) and Computer Science (25%) constitute the dominant subsets, followed by Astronomy (13%), Biology (10%), and Mathematics (9%). The 'Others' category (10%) encompasses diverse fields such as Engineering and Earth Science.
• Figure 2: Character Length Distribution Analysis. Comparative statistics reveal a significant shift in information density between the raw and enhanced corpora. Raw captions (orange) exhibit high volatility with a character count of $267 \pm 261$ (mean $\pm$ std), reflecting the pervasive semantic sparsity and inconsistency inherent in original scientific metadata. In contrast, the semantically enhanced descriptions (blue) achieve a robust 2.8$\times$ expansion, yielding $759 \pm 251$ characters. The reduction in the coefficient of variation ($CV \approx 33\%$) for the recaptioned data indicates a more homogeneous and standardized semantic representation, effectively bridging the "semantic gap" for downstream multimodal training.
• Figure 3: Overview of the S1-MMAlign Data Construction Pipeline. The workflow consists of four distinct phases: (1) Data Ingestion from diverse sources including arXiv (LaTeX) and web crawls (PDF); (2) Preprocessing Pipeline, featuring archive integrity checks, Regex-based parsing, EPS-to-PNG conversion, and strict quality filtering; (3) Core AI Processing, employing the Qwen-VL architecture on an H100 GPU cluster to generate semantically dense captions; and (4) Structured Output Generation, organizing the final data into JSONL format for public release.
• Figure 4: File Organization of S1-MMAlign. The repository structure adapts to data volume: (A) Yearly Archives for massive sources like arXiv (e.g., images_2007.tar.gz); (B) Multi-Part Archives for large sources like bioRxiv (e.g., images.tar.gz.partaa); and (C) Single Archives for smaller datasets. All subsets include a jsonl directory for metadata.
• Figure 5: Empirical CDF of Text Quality (Pseudo-PPL). The plot illustrates the Cumulative Distribution Function of $\log_{10}(\text{pseudo-PPL})$ scores derived from SciBERT beltagy2019scibert. The blue curve (Enhanced Captions) demonstrates a pronounced leftward shift compared to the original captions, confirming a significant reduction in perplexity and superior alignment with scientific linguistic norms.