GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-training

TL;DR

GeoX tackles automatic geometry problem solving by introducing a formalized vision-language pre-training framework that unifies geometry diagrams and symbolic language into a coherent solving pipeline. It combines unimodal pre-training with a geometry-focused encoder and symbol decoder, geometry-language alignment via GS-Former, and end-to-end visual instruction tuning to produce verifiable solution programs that are executed by a solver. Empirical results across GeoQA, UniGeo, Geometry3K, PGPS9K, and MathVista-GEO demonstrate state-of-the-art performance and improved interpretability through formalized reasoning steps. This work highlights the value of formalized representations for geometry and lays groundwork for geometry-focused generalist models with strong cross-modal grounding and verifiable reasoning.

Abstract

Despite their proficiency in general tasks, Multi-modal Large Language Models (MLLMs) struggle with automatic Geometry Problem Solving (GPS), which demands understanding diagrams, interpreting symbols, and performing complex reasoning. This limitation arises from their pre-training on natural images and texts, along with the lack of automated verification in the problem-solving process. Besides, current geometric specialists are limited by their task-specific designs, making them less effective for broader geometric problems. To this end, we present GeoX, a multi-modal large model focusing on geometric understanding and reasoning tasks. Given the significant differences between geometric diagram-symbol and natural image-text, we introduce unimodal pre-training to develop a diagram encoder and symbol decoder, enhancing the understanding of geometric images and corpora. Furthermore, we introduce geometry-language alignment, an effective pre-training paradigm that bridges the modality gap between unimodal geometric experts. We propose a Generator-And-Sampler Transformer (GS-Former) to generate discriminative queries and eliminate uninformative representations from unevenly distributed geometric signals. Finally, GeoX benefits from visual instruction tuning, empowering it to take geometric images and questions as input and generate verifiable solutions. Experiments show that GeoX outperforms both generalists and geometric specialists on publicly recognized benchmarks, such as GeoQA, UniGeo, Geometry3K, and PGPS9k.

Figures (7)

• Figure 1: Highlights of GeoX: 1) Comparison between GPT-4V openai2023gpt4v and GeoX: GPT-4V often fails to provide the expected results or solving approaches. Besides, verifying GPT-4V’s solutions is labor-intensive, requiring expert knowledge and step-by-step analysis. 2) Comparison between formal and natural (informal) language: Unlike existing works g-llava/gao2023gmavis/zhang2024mavis that use natural language, we advocate for formal language due to its effectiveness and verifiability, making it more suitable for geometric tasks. 3) GeoX solves geometric tasks in a unified format by taking geometric images and questions as input, generating verifiable program sequences, and performing solving with a solver.
• Figure 2: Overview of GeoX for training. We present a versatile method for automatic geometric problem solving through unified formalized vision-language pre-training, which comprises three progressive stages.
• Figure 3: Effectiveness of Uni-modal Pre-training. We compare the widely used CLIP-ViT-B and our Geo-ViT-B, along with three LLM models: LLAMA-2-7B, LLEMMA-7B, and our Geo-LLM-7B.
• Figure 4: Visualization results on four datasets by our GeoX.
• Figure 5: Four visualized examples of geometric problem in natural images solved by our GeoX.