Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do Large Language Models Truly Understand Geometric Structures?

Xiaofeng Wang, Yiming Wang, Wenhong Zhu, Rui Wang

TL;DR

This work introduces GeomRel, a dedicated benchmark that isolates geometric Relationship Identification (GRI) to more accurately assess whether LLMs truly understand geometric structures, revealing substantial weaknesses in current models, especially for angle-based relations. To address these gaps, the authors propose GeoCoT, a two-stage Geometry Chain-of-Thought prompting approach that first breaks down geometry into fundamental elements and then applies reverse reasoning to infer relations, yielding notable improvements across basic and advanced datasets. Through broad evaluations across nine LLMs, various data-diversity operations, and prompting strategies, the paper demonstrates that end-to-end accuracy can obscure underlying deficits in geometric understanding and that explicit, structured reasoning substantially boosts performance. The findings suggest a practical direction for enhancing LLM geometric reasoning, with GeoCoT offering a scalable method to improve model reliability in geometry-related tasks.

Abstract

Geometric ability is a significant challenge for large language models (LLMs) due to the need for advanced spatial comprehension and abstract thinking. Existing datasets primarily evaluate LLMs on their final answers, but they cannot truly measure their true understanding of geometric structures, as LLMs can arrive at correct answers by coincidence. To fill this gap, we introduce the GeomRel dataset, designed to evaluate LLMs' understanding of geometric structures by isolating the core step of geometric relationship identification in problem-solving. Using this benchmark, we conduct thorough evaluations of diverse LLMs and identify key limitations in understanding geometric structures. We further propose the Geometry Chain-of-Thought (GeoCoT) method, which enhances LLMs' ability to identify geometric relationships, resulting in significant performance improvements.

Do Large Language Models Truly Understand Geometric Structures?

TL;DR

This work introduces GeomRel, a dedicated benchmark that isolates geometric Relationship Identification (GRI) to more accurately assess whether LLMs truly understand geometric structures, revealing substantial weaknesses in current models, especially for angle-based relations. To address these gaps, the authors propose GeoCoT, a two-stage Geometry Chain-of-Thought prompting approach that first breaks down geometry into fundamental elements and then applies reverse reasoning to infer relations, yielding notable improvements across basic and advanced datasets. Through broad evaluations across nine LLMs, various data-diversity operations, and prompting strategies, the paper demonstrates that end-to-end accuracy can obscure underlying deficits in geometric understanding and that explicit, structured reasoning substantially boosts performance. The findings suggest a practical direction for enhancing LLM geometric reasoning, with GeoCoT offering a scalable method to improve model reliability in geometry-related tasks.

Abstract

Geometric ability is a significant challenge for large language models (LLMs) due to the need for advanced spatial comprehension and abstract thinking. Existing datasets primarily evaluate LLMs on their final answers, but they cannot truly measure their true understanding of geometric structures, as LLMs can arrive at correct answers by coincidence. To fill this gap, we introduce the GeomRel dataset, designed to evaluate LLMs' understanding of geometric structures by isolating the core step of geometric relationship identification in problem-solving. Using this benchmark, we conduct thorough evaluations of diverse LLMs and identify key limitations in understanding geometric structures. We further propose the Geometry Chain-of-Thought (GeoCoT) method, which enhances LLMs' ability to identify geometric relationships, resulting in significant performance improvements.
Paper Structure (45 sections, 1 equation, 13 figures, 29 tables, 1 algorithm)

This paper contains 45 sections, 1 equation, 13 figures, 29 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. dataset: Geometric Relationship Identification Benchmark
  3. Finite Geometric Relationship Pool
  4. Basic Data Generation
  5. Advanced Data Generation
  6. Indeterminate Data Generation
  7. Increasing Data Diversity
  8. Comprehensive Evaluation
  9. Experimental Setup
  10. Main Results (Table \ref{['tab:mainresults']})
  11. Performances on Basic dataset.
  12. Performances on Advanced dataset.
  13. Angle-based Relations are Particularly Difficult for LLMs.
  14. Evaluation Reasonableness.
  15. Bias between geometric relations that LLMs subjectively fail to identify and those that are objectively unidentifiable.
  16. ...and 30 more sections

Figures (13)

  • Figure 1: The general process of solving geometric problems (Middle). Within this process, identifying the geometric relationships is a fundamental step, one must first accurately identify the geometric structures, then apply theorems for reasoning or calculation to reach the final answer. Compared to traditional geometric datasets that only assess the accuracy of final answers (Top), we extract the fundamental steps of geometric relationship identification (GRI) to create the $\mathsf{GeomRel}$ benchmark, which evaluates whether LLMs truly understand geometric structures (Bottom).
  • Figure 2: The dataset Framework
  • Figure 3: Accuracy correlations between basic and advanced dataset.
  • Figure 4: Ablation study about data diversity strategies of our dataset on GPT-3.5-Turbo model.
  • Figure 5: Comparison of Models
  • ...and 8 more figures