PRiSM: An Agentic Multimodal Benchmark for Scientific Reasoning via Python-Grounded Evaluation

TL;DR

PRiSM addresses limitations in vision-language benchmarks for scientific reasoning by delivering a dynamic, multimodal dataset with executable Python ground truth. Using the PrismAgent pipeline, it generates 24,750 university-level physics and math problems with parameterization, paraphrase variants, and ground-truth code for verification. The benchmark defines five diagnostic tasks—generalization, symbolic program synthesis, perturbation robustness, reasoning correction, and ambiguity resolution—to enable fine-grained auditing of multimodal reasoning. Findings indicate current VLMs struggle with robustness, multi-step reasoning, and uncertainty handling, underscoring the need for structured verification and tool-enabled evaluation in scientific contexts.

Abstract

Evaluating vision-language models (VLMs) in scientific domains like mathematics and physics poses unique challenges that go far beyond predicting final answers. These domains demand conceptual understanding, symbolic reasoning, and adherence to formal laws, requirements that most existing benchmarks fail to address. In particular, current datasets tend to be static, lacking intermediate reasoning steps, robustness to variations, or mechanisms for verifying scientific correctness. To address these limitations, we introduce PRiSM, a synthetic, fully dynamic, and multimodal benchmark for evaluating scientific reasoning via grounded Python code. PRiSM includes over 24,750 university-level physics and math problems, and it leverages our scalable agent-based pipeline, PrismAgent, to generate well-structured problem instances. Each problem contains dynamic textual and visual input, a generated figure, alongside rich structured outputs: executable Python code for ground truth generation and verification, and detailed step-by-step reasoning. The dynamic nature and Python-powered automated ground truth generation of our benchmark allow for fine-grained experimental auditing of multimodal VLMs, revealing failure modes, uncertainty behaviors, and limitations in scientific reasoning. To this end, we propose five targeted evaluation tasks covering generalization, symbolic program synthesis, perturbation robustness, reasoning correction, and ambiguity resolution. Through comprehensive evaluation of existing VLMs, we highlight their limitations and showcase how PRiSM enables deeper insights into their scientific reasoning capabilities.

Figures (9)

• Figure 1: An instance from the PRiSM dataset includes a parameterized question with substituted input variables, a figure generated with problem inputs, a step-by-step solution, and the corresponding Python code.
• Figure 2: Examples from the PRiSM dataset illustrating Task I (Robustness to Input Variations) and Task III (Reasoning with Correction). For Task I, we vary input values and paraphrase the problem text to evaluate numerical generalization. For Task III, we introduce different types of incorrect reasoning steps and assess the model's ability to detect and correct mistakes in multi-step solutions.
• Figure 3: Overview of the Dataset Creation Pipeline Enabled by Our Framework.
• Figure 4: Illustration of modality conflict where the model’s correct algebraic solution is overridden by a misleading visual cue.
• Figure 5: Example of ambiguity in handwritten digits leading to misinterpretation.