Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluation

Yuan Feng, Yue Yang, Xiaohan He, Jiatong Zhao, Jianlong Chen, Zijun Chen, Daocheng Fu, Qi Liu, Renqiu Xia, Bo Zhang, Junchi Yan

TL;DR

GeoBench tackles the inadequacy of existing geometric problem-solving benchmarks by introducing a four-level hierarchical framework that separately probes visual perception, planning, theorem application, and self-reflective backtracking. Built on TrustGeoGen-generated problems, it yields 1,021 synthetic, verifiably solvable samples across six tasks, enabling fine-grained diagnostics beyond end results. The study reveals that sub-goal decomposition and premise filtering are critical for complex GPS performance, while chain-of-thought prompting can impair specific tasks such as faulty branch localization. Overall, GeoBench demonstrates robust generalization and provides actionable guidance for building more capable and verifiable geometric reasoning systems, with future work extending to 3D geometry and increased data diversity.

Abstract

Geometric problem solving constitutes a critical branch of mathematical reasoning, requiring precise analysis of shapes and spatial relationships. Current evaluations of geometric reasoning in vision-language models (VLMs) face limitations, including the risk of test data contamination from textbook-based benchmarks, overemphasis on final answers over reasoning processes, and insufficient diagnostic granularity. To address these issues, we present GeoBench, a hierarchical benchmark featuring four reasoning levels in geometric problem-solving: Visual Perception, Goal-Oriented Planning, Rigorous Theorem Application, and Self-Reflective Backtracking. Through six formally verified tasks generated via TrustGeoGen, we systematically assess capabilities ranging from attribute extraction to logical error correction. Experiments reveal that while reasoning models like OpenAI-o3 outperform general MLLMs, performance declines significantly with increasing task complexity. Key findings demonstrate that sub-goal decomposition and irrelevant premise filtering critically influence final problem-solving accuracy, whereas Chain-of-Thought prompting unexpectedly degrades performance in some tasks. These findings establish GeoBench as a comprehensive benchmark while offering actionable guidelines for developing geometric problem-solving systems.

GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluation

TL;DR

GeoBench tackles the inadequacy of existing geometric problem-solving benchmarks by introducing a four-level hierarchical framework that separately probes visual perception, planning, theorem application, and self-reflective backtracking. Built on TrustGeoGen-generated problems, it yields 1,021 synthetic, verifiably solvable samples across six tasks, enabling fine-grained diagnostics beyond end results. The study reveals that sub-goal decomposition and premise filtering are critical for complex GPS performance, while chain-of-thought prompting can impair specific tasks such as faulty branch localization. Overall, GeoBench demonstrates robust generalization and provides actionable guidance for building more capable and verifiable geometric reasoning systems, with future work extending to 3D geometry and increased data diversity.

Abstract

Geometric problem solving constitutes a critical branch of mathematical reasoning, requiring precise analysis of shapes and spatial relationships. Current evaluations of geometric reasoning in vision-language models (VLMs) face limitations, including the risk of test data contamination from textbook-based benchmarks, overemphasis on final answers over reasoning processes, and insufficient diagnostic granularity. To address these issues, we present GeoBench, a hierarchical benchmark featuring four reasoning levels in geometric problem-solving: Visual Perception, Goal-Oriented Planning, Rigorous Theorem Application, and Self-Reflective Backtracking. Through six formally verified tasks generated via TrustGeoGen, we systematically assess capabilities ranging from attribute extraction to logical error correction. Experiments reveal that while reasoning models like OpenAI-o3 outperform general MLLMs, performance declines significantly with increasing task complexity. Key findings demonstrate that sub-goal decomposition and irrelevant premise filtering critically influence final problem-solving accuracy, whereas Chain-of-Thought prompting unexpectedly degrades performance in some tasks. These findings establish GeoBench as a comprehensive benchmark while offering actionable guidelines for developing geometric problem-solving systems.
Paper Structure (41 sections, 3 equations, 7 figures, 13 tables)

This paper contains 41 sections, 3 equations, 7 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work and Preliminaries
  3. Methodology
  4. Input Data Preparation
  5. Hierarchical Evaluation Framework
  6. Level-1: Visual Perception
  7. Numerical Perception.
  8. Structural Perception.
  9. Level-2: Goal-Oriented Planning
  10. Irrelevant Premise Filtering.
  11. Sub-Goal Decomposition.
  12. Level-3: Rigorous Theorem Application
  13. Theorem Selection.
  14. Level-4: Self-Reflective Backtracking
  15. Faulty Branch Localization.
  16. ...and 26 more sections

Figures (7)

  • Figure 1: Overview of GeoBench: (1) Generation of formally verified geometry problems with images, problems and reasoning graphs via TrustGeoGen trustgeogen2025, followed by (2) systematic construction of hierarchical evaluation tasks through four reasoning capability levels.
  • Figure 2: Data Distribution Comparison between GeoBench and GeoQA geoqa_ngs/chen2021geoqa.
  • Figure 3: Performance comparison w/ & w/o Chain-of-Thought (CoT) across six tasks in GeoBench.
  • Figure 4: Failure cases (some question descriptions and prediction content omitted for space limit).
  • Figure 5: Prompts Format
  • ...and 2 more figures