Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utility

TL;DR

This work benchmarks scientific image synthesis across pixel-based and programmatic generations, introducing ImgCoder as a logic-driven alternative and SciGenBench as a specialized, reasoning-oriented benchmark. It reveals a precision–expressiveness trade-off where code-based methods excel at structural correctness while pixel-based methods provide richer visuals, and shows that fine-tuning large multimodal models on rigorously verified synthetic images yields reliable improvements in downstream multimodal reasoning. The paper also documents a distributional gap between synthetic and real scientific images, and demonstrates scaling laws indicating data-efficient gains with high-quality synthetic data. Overall, high-fidelity, logic-grounded synthesis presents a viable path to scaling multimodal reasoning in STEM.

Abstract

While synthetic data has proven effective for improving scientific reasoning in the text domain, multimodal reasoning remains constrained by the difficulty of synthesizing scientifically rigorous images. Existing Text-to-Image (T2I) models often produce outputs that are visually plausible yet scientifically incorrect, resulting in a persistent visual-logic divergence that limits their value for downstream reasoning. Motivated by recent advances in next-generation T2I models, we conduct a systematic study of scientific image synthesis across generation paradigms, evaluation, and downstream use. We analyze both direct pixel-based generation and programmatic synthesis, and propose ImgCoder, a logic-driven framework that follows an explicit "understand - plan - code" workflow to improve structural precision. To rigorously assess scientific correctness, we introduce SciGenBench, which evaluates generated images based on information utility and logical validity. Our evaluation reveals systematic failure modes in pixel-based models and highlights a fundamental expressiveness-precision trade-off. Finally, we show that fine-tuning Large Multimodal Models (LMMs) on rigorously verified synthetic scientific images yields consistent reasoning gains, with potential scaling trends analogous to the text domain, validating high-fidelity scientific synthesis as a viable path to unlocking massive multimodal reasoning capabilities.

Figures (19)

• Figure 1: Methodological Overview. The framework consists of three core components: (1) Scientific Image Generation (Left), where we propose ImgCoder, a programmatic approach decoupling planning from implementation to outperform pixel-based baselines; (2) SciGenBench Construction (Top Right), a rigorously curated benchmark with a fine-grained taxonomy and atomic quizzes; and (3) Evaluation Framework (Bottom Right), a multi-faceted assessment system combining LMM judges, inverse validation, standard metrics, and downstream performance.
• Figure 2: Precision vs. Expressiveness Trade-off.Left (a): When plotting the function $y=x \ln x$, pixel-based models produce visually smooth but mathematically inaccurate plots, while code-based methods ensure exactness via execution. Right (b): Conversely, for physical scenarios like a spring system, pixel-based models offer richer visual expressiveness, whereas code-based outputs remain schematic.
• Figure 3: Qualitative Error Taxonomy. We categorize failures into five modes ranging from low-level visual artifacts to high-level semantic hallucinations. The specific errors are annotated in red.
• Figure 4: Analysis of Distributional Gaps.Left: CLIP t-SNE shows clear separation between NanoBanana-Pro and real images. Right: Spectral analysis attributes this gap to excessive high-frequency energy in generated images.
• Figure 5: Downstream Data Utility.Left (a): Stronger teachers yield higher training rewards. Middle (b): Filtered data consistently outperforms unfiltered data. Right (c): Performance scales predictably with data size.