Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Multimodal LLMs Ability In Geometry Problem Solving, Reasoning, And Multistep Scoring

Avinash Anand, Raj Jaiswal, Abhishek Dharmadhikari, Atharva Marathe, Harsh Parimal Popat, Harshil Mital, Kritarth Prasad, Rajiv Ratn Shah, Roger Zimmermann

TL;DR

This work introduces GPSM4K, a comprehensive multimodal geometry dataset with roughly 4,000 problems (expanded to 5,340) that includes numerical and theorem-proving questions and two solution formats to enable detailed reasoning analysis. The authors develop a data-augmentation pipeline (diagram descriptions, question generation, and solution regeneration) and evaluate LVLMs with zero-shot and fine-tuning experiments, image captioning, and multimodal RAG. Key findings show that fine-tuning on GPSM4K improves geometry reasoning, larger models like GPT-4 and Gemini-Pro perform best in zero-shot settings, and caption quality and RAG contribute meaningful gains, though open-source models still lag on theorem-based tasks. The study highlights the importance of rich, step-by-step explanations and multimodal context for advancing geometric reasoning in LVLMs, and it outlines future directions to close remaining gaps through richer theorem content and improved caption generation.

Abstract

This paper presents GPSM4K, a comprehensive geometry multimodal dataset tailored to augment the problem-solving capabilities of Large Vision Language Models (LVLMs). GPSM4K encompasses 2157 multimodal question-answer pairs manually extracted from mathematics textbooks spanning grades 7-12 and is further augmented to 5340 problems, consisting of both numerical and theorem-proving questions. In contrast to PGPS9k, Geometry3K, and Geo170K which feature only objective-type questions, GPSM4K offers detailed step-by-step solutions in a consistent format, facilitating a comprehensive evaluation of problem-solving approaches. This dataset serves as an excellent benchmark for assessing the geometric reasoning capabilities of LVLMs. Evaluation of our test set shows that there is scope for improvement needed in open-source language models in geometry problem-solving. Finetuning on our training set increases the geometry problem-solving capabilities of models. Further, We also evaluate the effectiveness of techniques such as image captioning and Retrieval Augmentation generation (RAG) on model performance. We leveraged LLM to automate the task of final answer evaluation by providing ground truth and predicted solutions. This research will help to assess and improve the geometric reasoning capabilities of LVLMs.

Improving Multimodal LLMs Ability In Geometry Problem Solving, Reasoning, And Multistep Scoring

TL;DR

This work introduces GPSM4K, a comprehensive multimodal geometry dataset with roughly 4,000 problems (expanded to 5,340) that includes numerical and theorem-proving questions and two solution formats to enable detailed reasoning analysis. The authors develop a data-augmentation pipeline (diagram descriptions, question generation, and solution regeneration) and evaluate LVLMs with zero-shot and fine-tuning experiments, image captioning, and multimodal RAG. Key findings show that fine-tuning on GPSM4K improves geometry reasoning, larger models like GPT-4 and Gemini-Pro perform best in zero-shot settings, and caption quality and RAG contribute meaningful gains, though open-source models still lag on theorem-based tasks. The study highlights the importance of rich, step-by-step explanations and multimodal context for advancing geometric reasoning in LVLMs, and it outlines future directions to close remaining gaps through richer theorem content and improved caption generation.

Abstract

This paper presents GPSM4K, a comprehensive geometry multimodal dataset tailored to augment the problem-solving capabilities of Large Vision Language Models (LVLMs). GPSM4K encompasses 2157 multimodal question-answer pairs manually extracted from mathematics textbooks spanning grades 7-12 and is further augmented to 5340 problems, consisting of both numerical and theorem-proving questions. In contrast to PGPS9k, Geometry3K, and Geo170K which feature only objective-type questions, GPSM4K offers detailed step-by-step solutions in a consistent format, facilitating a comprehensive evaluation of problem-solving approaches. This dataset serves as an excellent benchmark for assessing the geometric reasoning capabilities of LVLMs. Evaluation of our test set shows that there is scope for improvement needed in open-source language models in geometry problem-solving. Finetuning on our training set increases the geometry problem-solving capabilities of models. Further, We also evaluate the effectiveness of techniques such as image captioning and Retrieval Augmentation generation (RAG) on model performance. We leveraged LLM to automate the task of final answer evaluation by providing ground truth and predicted solutions. This research will help to assess and improve the geometric reasoning capabilities of LVLMs.

Paper Structure

This paper contains 34 sections, 12 equations, 9 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Geometry Multimodal Datasets
  4. PGPS9K
  5. Geoemtry3K
  6. Geo170K
  7. Multimodal LLMs
  8. LLaVA
  9. GLLaVA
  10. Retrieval Augmentated Generation
  11. Small LLMs vs Large LLMs
  12. Geometry Problem Solving
  13. Data Generation via LLM
  14. Need for Dataset Creation
  15. Methodology
  16. ...and 19 more sections

Figures (9)

  • Figure 1: Comparison of Geometry3K, PGPS9K, and our own GPSM4K datasets, showcasing geometric problems and their representations of complexity in solutions.
  • Figure 2: Dataset augmentation Pipeline using the Gemini-Pro API. The process starts with the original question and solution, generates new questions with varied parameters, and regenerates detailed step-by-step solutions with equation expressions highlighted in bold. The augmentation enhances the dataset by introducing variability and detailed solutions.
  • Figure 3: Data Extraction and topic-wise distribution
  • Figure 4: Prompt used for Solution Regeneration
  • Figure 5: Image captioning example
  • ...and 4 more figures