Toward an Artificial General Teacher: Procedural Geometry Data Generation and Visual Grounding with Vision-Language Models

Abstract

We study visual explanation in geometry education as a Referring Image Segmentation (RIS) problem: given a diagram and a natural language description, the task is to produce a pixel-level mask for the referred geometric element. However, existing RIS models trained on natural image benchmarks such as RefCOCO fail catastrophically on geometric diagrams due to the fundamental domain shift between photographic scenes and abstract, textureless schematics. To address the absence of suitable training data, we present a fully automated procedural data engine that generates over 200,000 synthetic geometry diagrams with pixel-perfect segmentation masks and linguistically diverse referring expressions, requiring zero manual annotation. We further propose domain-specific fine-tuning of vision-language models (VLMs), demonstrating that a fine-tuned Florence-2 achieves 49% IoU and 85% Buffered IoU (BIoU), compared to <1% IoU in zero-shot settings. We introduce Buffered IoU, a geometry-aware evaluation metric that accounts for thin-structure localization, and show that it better reflects true segmentation quality than standard IoU. Our results establish a foundation for building Artificial General Teachers (AGTs) capable of providing visually grounded, step-by-step explanations of geometry problems.

Figures (6)

• Figure 1: Overview. We formalize the "pointing" mechanism of human geometry teaching as a Referring Image Segmentation (RIS) task. Given a diagram and a textual description (e.g., "triangle ABC"), the model produces a pixel-level mask highlighting the referred element.
• Figure 2: Data generation pipeline. Stage 1: Constraint-based coordinate generation via analytical solvers. Stage 2: Vector rendering through LaTeX/TikZ templates. Stage 3: Render-based ground-truth mask extraction using channel arithmetic.
• Figure 3: Hierarchical language template system. Referring expressions are generated at multiple levels of linguistic complexity to encourage robust visual grounding.
• Figure 4: VLM fine-tuning architecture with polygon-based RIS. The vision-language model employs LoRA adapters (shown in red) for efficient fine-tuning while keeping the vision encoder, cross-attention fusion and Token Decoder weights frozen (shown in purple). The model autoregressively generates coordinate tokens that are rasterized into binary masks.
• Figure 5: Qualitative results. The fine-tuned Florence-2 model accurately segments lines, triangles, and circles across diverse geometric queries and diagrammatic complexities.