Do Multi-Agents Solve Better Than Single? Evaluating Agentic Frameworks for Diagram-Grounded Geometry Problem Solving and Reasoning

TL;DR

The paper investigates whether decomposing diagram-grounded geometry problem solving into multiple agents (an interpreter to extract symbolic literals and a solver to reason over them) offers advantages over a single-end-to-end model. It benchmarks both paradigms across Geometry3K, MathVerse, OlympiadBench, and We-Math using open-source and closed-source models, revealing that multi-agent pipelines provide clear gains for open-source systems while strong proprietary models often perform better in single-agent mode, especially on classic benchmarks; gains on newer datasets are more modest and tied to the quality of the interpreter predicates. A predicate-alignment analysis shows that the semantic fidelity of literals strongly influences downstream reasoning, with higher-quality interpreters yielding better solver performance. The work suggests adaptive strategies that switch between single- and multi-agent configurations based on model capacity and task characteristics, and provides code and data at the authors’ GitHub repository for replication and extension.

Abstract

Diagram-grounded geometry problem solving is a critical benchmark for multimodal large language models (MLLMs), yet the benefits of multi-agent design over single-agent remain unclear. We systematically compare single-agent and multi-agent pipelines on four visual math benchmarks: Geometry3K, MathVerse, OlympiadBench, and We-Math. For open-source models, multi-agent consistently improves performance. For example, Qwen-2.5-VL (7B) gains +6.8 points and Qwen-2.5-VL (32B) gains +3.3 on Geometry3K, and both Qwen-2.5-VL variants see further gains on OlympiadBench and We-Math. In contrast, the closed-source Gemini-2.0-Flash generally performs better in single-agent mode on classic benchmarks, while multi-agent yields only modest improvements on the newer We-Math dataset. These findings show that multi-agent pipelines provide clear benefits for open-source models and can assist strong proprietary systems on newer, less familiar benchmarks, but agentic decomposition is not universally optimal. All code, data, and reasoning files are available at https://github.com/faiyazabdullah/Interpreter-Solver

Figures (1)

• Figure 1: (a) An Interpreter Agent generates formal predicates from images and questions using VLMs. (b) A Solver Agent then solves the problem using these predicates as LLM input.