Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DynaSolidGeo: A Dynamic Benchmark for Genuine Spatial Mathematical Reasoning of VLMs in Solid Geometry

Changti Wu, Shijie Lian, Zihao Liu, Lei Zhang, Laurence Tianruo Yang, Kai Chen

TL;DR

This work addresses the challenge of measuring genuine spatial mathematical reasoning in Vision-Language Models for solid geometry. It introduces DynaSolidGeo, a dynamic benchmark with 503 seed questions parameterizable into unbounded multimodal instances, rendering either randomized view images or 360-degree videos. Evaluation uses three metrics ($AA$, $PS$, $PA$) and a semi-automatic data annotation pipeline; experiments show large gaps between model families, notable degradation under dynamic evaluation, and evidence of data contamination on static sources. The benchmark mitigates memorization through dynamic generation and promotes progress in process-grounded reasoning for spatial problems.

Abstract

Solid geometry problem solving demands spatial mathematical reasoning that integrates spatial intelligence and symbolic reasoning. However, most existing multimodal mathematical reasoning benchmarks focus primarily on 2D plane geometry, rely on static datasets prone to data contamination and memorization, and evaluate models solely by final answers, overlooking the reasoning process. To address these limitations, we introduce DynaSolidGeo, the first dynamic benchmark for evaluating genuine spatial reasoning in Vision-Language Models (VLMs). Constructed through a semi-automatic annotation pipeline, DynaSolidGeo contains 503 expert-curated seed questions that can, in principle, dynamically generate an unbounded number of diverse multimodal text-visual instances. Beyond answer accuracy, we incorporate process evaluation based on expert-annotated reasoning chains to measure logical validity and causal coherence. Experiments across representative open-source and closed-source VLMs reveal large performance gaps, severe degradation in dynamic settings, and poor performance on tasks requiring high-level spatial intelligence, such as mental rotation and visualization. The code and dataset are available at \href{https://zgca-ai4edu.github.io/DynaSolidGeo/}{DynaSolidGeo}.

DynaSolidGeo: A Dynamic Benchmark for Genuine Spatial Mathematical Reasoning of VLMs in Solid Geometry

TL;DR

This work addresses the challenge of measuring genuine spatial mathematical reasoning in Vision-Language Models for solid geometry. It introduces DynaSolidGeo, a dynamic benchmark with 503 seed questions parameterizable into unbounded multimodal instances, rendering either randomized view images or 360-degree videos. Evaluation uses three metrics (, , ) and a semi-automatic data annotation pipeline; experiments show large gaps between model families, notable degradation under dynamic evaluation, and evidence of data contamination on static sources. The benchmark mitigates memorization through dynamic generation and promotes progress in process-grounded reasoning for spatial problems.

Abstract

Solid geometry problem solving demands spatial mathematical reasoning that integrates spatial intelligence and symbolic reasoning. However, most existing multimodal mathematical reasoning benchmarks focus primarily on 2D plane geometry, rely on static datasets prone to data contamination and memorization, and evaluate models solely by final answers, overlooking the reasoning process. To address these limitations, we introduce DynaSolidGeo, the first dynamic benchmark for evaluating genuine spatial reasoning in Vision-Language Models (VLMs). Constructed through a semi-automatic annotation pipeline, DynaSolidGeo contains 503 expert-curated seed questions that can, in principle, dynamically generate an unbounded number of diverse multimodal text-visual instances. Beyond answer accuracy, we incorporate process evaluation based on expert-annotated reasoning chains to measure logical validity and causal coherence. Experiments across representative open-source and closed-source VLMs reveal large performance gaps, severe degradation in dynamic settings, and poor performance on tasks requiring high-level spatial intelligence, such as mental rotation and visualization. The code and dataset are available at \href{https://zgca-ai4edu.github.io/DynaSolidGeo/}{DynaSolidGeo}.
Paper Structure (37 sections, 3 equations, 6 figures, 10 tables)

This paper contains 37 sections, 3 equations, 6 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Mathematical Reasoning Benchmarks.
  4. Vision-Language Models.
  5. DynaSolidGeo
  6. Data Collection
  7. Data Annotation Pipeline
  8. Statistics
  9. Evaluation Metrics
  10. Answer Accuracy
  11. Process Score
  12. Process-Qualified Accuracy
  13. Experiment
  14. Experimental Setup
  15. Experimental Results
  16. ...and 22 more sections

Figures (6)

  • Figure 1: Overview of the data annotation pipeline and the application of seed questions. Annotation: 1) Expert-Guided Parametrization and Visualization: Each source question is first parameterized into a JSON annotation and paired with a MATLAB visualization program. 2) Automated Python Program Synthesis: The pipeline then synthesizes parameterized Python programs that generate textual descriptions and MATLAB invoke commands. 3) Automated MATLAB Program Synthesis: Correspondingly, the pipeline then synthesizes the parameterized version of MATLAB programs for figure and video rendering. 4) Expert Verification: Final human checks ensure the correctness and usability of seed questions. Application: By inputting a random seed, each seed question is instantiated into a question instance.
  • Figure 2: Distribution of variable parameters per seed question.
  • Figure 3: Comparison of model performance on Answer Accuracy (AA) between DynaSolidGeo and source questions.
  • Figure 4: Error analysis.
  • Figure 5: Detailed error distribution on Gemini-2.5-Pro.
  • ...and 1 more figures