Socratic-Geo: Synthetic Data Generation and Geometric Reasoning via Multi-Agent Interaction

TL;DR

The work tackles the data bottleneck in geometric reasoning for multimodal models by introducing Socratic-Geo, a fully autonomous, goal-driven data-synthesis framework that couples generation with learning through a triad of agents (Teacher, Solver, Generator) and validation via Reflect and RePI. The Teacher crafts executable geometry code with verification loops, the Solver learns via Group Relative Policy Optimization guided by failure-driven augmentation, and the Generator distills the Teacher's drawing rules into a diffusion model trained on image-code-instruction pairs. From 108 seed problems, the approach reaches $49.11$ accuracy across six benchmarks using roughly one-quarter of baseline data, while the open-source Socratic-Generator achieves $42.4$ on GenExam-Math, setting a new state-of-the-art and approaching closed-source systems. The results demonstrate strong data efficiency, domain generalization to chart reasoning and multimodal coding, and a scalable, self-improving data engine for STEM reasoning tasks.

Abstract

Multimodal Large Language Models (MLLMs) have significantly advanced vision-language understanding. However, even state-of-the-art models struggle with geometric reasoning, revealing a critical bottleneck: the extreme scarcity of high-quality image-text pairs. Human annotation is prohibitively expensive, while automated methods fail to ensure fidelity and training effectiveness. Existing approaches either passively adapt to available images or employ inefficient random exploration with filtering, decoupling generation from learning needs. We propose Socratic-Geo, a fully autonomous framework that dynamically couples data synthesis with model learning through multi-agent interaction. The Teacher agent generates parameterized Python scripts with reflective feedback (Reflect for solvability, RePI for visual validity), ensuring image-text pair purity. The Solver agent optimizes reasoning through preference learning, with failure paths guiding Teacher's targeted augmentation. Independently, the Generator learns image generation capabilities on accumulated "image-code-instruction" triplets, distilling programmatic drawing intelligence into visual generation. Starting from only 108 seed problems, Socratic-Solver achieves 49.11 on six benchmarks using one-quarter of baseline data, surpassing strong baselines by 2.43 points. Socratic-Generator achieves 42.4% on GenExam, establishing new state-of-the-art for open-source models, surpassing Seedream-4.0 (39.8%) and approaching Gemini-2.5-Flash-Image (43.1%).

Figures (3)

• Figure 1: Overall performance comparison demonstrating the dual effectiveness of the Socratic-Geo framework in both reasoning and generation. (a) Our Socratic-Generator-Image achieves an impressive 42.4 Relaxed score on the GenExam-Math benchmark, establishing a new state-of-the-art for open-source models and matches strong closed-source systems like Gemini-2.5-Flash-Image. (b) Our Socratic-Solver achieves an impressive 49.11% average accuracy across the reasoning benchmarks, marking a substantial +4.13 point improvement over the zero-shot baseline and consistently outperforming all other fine-tuning methods.
• Figure 2: Overview of the Socratic-Geo framework. (a) A closed-loop reasoning process where the Solver’s failures trigger the Teacher to invent and validate new problems, enriching the curriculum. (b) The Generator distills the Teacher’s programmatic drawing instructions into a standalone image synthesis model.
• Figure 3: A concrete instantiation of the Socratic-Geo interaction pipeline, where the Teacher transforms flawed reasoning into diagnostic problems. Red highlighted regions mark critical intervention points including error diagnosis and geometric augmentation. Left: The Solver incorrectly assumes a right triangle structure, overlooking the given constraint $\angle BAC = 60^\circ$, leading to an invalid solution path. Right: The Teacher introduces point $P$, forcing the Solver to apply the inscribed angle theorem combined with the $60^\circ$ angle property to establish the relationship between $\angle APD$ and $\angle ACD$, directly targeting the reasoning deficiency.