NoReGeo: Non-Reasoning Geometry Benchmark

TL;DR

NoReGeo introduces a cross-modal geometry benchmark with $2{,}500$ items across $25$ categories to assess intrinsic geometric understanding in LLMs and VLMs without chain-of-thought. Across $45+$ models, full visual context boosts VLM performance, yet top scores (~$65{.}0 ext{%}$) lag human baselines (~$74{.}5 ext{%}$) and reveal limited native geometric perception in current architectures. Crucially, a simple linear probe on frozen vision encoders nearly solves the tasks, implying that geometric cues are latent in embeddings and not naturally accessed by standard LLM training. The work points to a need for training regimes and model architectures that cultivate true geometric cognition for rapid, perception-based spatial reasoning in real-time systems.

Abstract

We present NoReGeo, a novel benchmark designed to evaluate the intrinsic geometric understanding of large language models (LLMs) without relying on reasoning or algebraic computation. Unlike existing benchmarks that primarily assess models' proficiency in reasoning-based geometry-where solutions are derived using algebraic methods-NoReGeo focuses on evaluating whether LLMs can inherently encode spatial relationships and recognize geometric properties directly. Our benchmark comprises 2,500 trivial geometric problems spanning 25 categories, each carefully crafted to be solvable purely through native geometric understanding, assuming known object locations. We assess a range of state-of-the-art models on NoReGeo, including frontier models like GPT-4, observing that even the most advanced systems achieve an overall maximum of 65% accuracy in binary classification tasks. Further, our ablation experiments demonstrate that such geometric understanding does not emerge through fine-tuning alone, indicating that effective training for geometric comprehension requires a specialized approach from the outset. Our findings highlight a significant gap in current LLMs' ability to natively grasp geometric concepts, providing a foundation for future research toward models with true geometric cognition.

Figures (8)

• Figure 1: Evaluation samples from NoReGeo benchmark. Each problem is shown in three formats -- (a) text‑only, (b) text with dotted-image (points only), and (c) text with full-image (points plus connecting lines) -- together with the golden answer (yellow) and the model’s prediction.
• Figure 2: Distribution of model-level performance gaps per task category, comparing 'text with full image' to 'text with dotted image' setups.
• Figure 3: Detailed performance evaluation of large language models (LLMs) on the NoReGeo benchmark using only text inputs. Subfigure (a) shows task-level classification accuracy across all task types, while subfigure (b) focuses on regression error (MSE) for numeric tasks.
• Figure 4: Detailed performance evaluation of large language models (LLMs) on the NoReGeo benchmark using 'text + dotted images' inputs. Subfigure (a) shows task-level classification accuracy across all task types, while subfigure (b) focuses on regression error (MSE) for numeric tasks.
• Figure 5: Detailed performance evaluation of large language models (LLMs) on the NoReGeo benchmark using 'text + full images' inputs. Subfigure (a) shows task-level classification accuracy across all task types, while subfigure (b) focuses on regression error (MSE) for numeric tasks.