Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions

TL;DR

This work tackles the gap in low-level visual perception (LLVP) for multimodal LLMs by creating the Geoperception benchmark to test precise 2D geometric transcription. It shows current MLLMs struggle with basic geometry annotations and relations, motivating a synthetic-data engine and curriculum-based training. Through an empirical design-space study, the authors find CNN-based visual encoders and curriculum learning yield substantial gains, leading to the Euclid family of MLLMs trained solely on synthetic geometry data that generalize to real geometry tasks. Euclid achieves up to $82.98\%$ on challenging tasks and outperforms leading closed- and open-source models on multiple Geoperception tasks, demonstrating the practical potential of synthetic data and progressive training for LLVP in robotics, medical imaging, and manufacturing domains.

Abstract

Multimodal large language models (MLLMs) have made rapid progress in recent years, yet continue to struggle with low-level visual perception (LLVP) -- particularly the ability to accurately describe the geometric details of an image. This capability is crucial for applications in areas such as robotics, medical image analysis, and manufacturing. In this paper, we first introduce Geoperception, a benchmark designed to evaluate an MLLM's ability to accurately transcribe 2D geometric information from an image. Using this benchmark, we demonstrate the limitations of leading MLLMs, and then conduct a comprehensive empirical study to explore strategies for improving their performance on geometric tasks. Our findings highlight the benefits of certain model architectures, training techniques, and data strategies, including the use of high-fidelity synthetic data and multi-stage training with a data curriculum. Notably, we find that a data curriculum enables models to learn challenging geometry understanding tasks which they fail to learn from scratch. Leveraging these insights, we develop Euclid, a family of models specifically optimized for strong low-level geometric perception. Although purely trained on synthetic multimodal data, Euclid shows strong generalization ability to novel geometry shapes. For instance, Euclid outperforms the best closed-source model, Gemini-1.5-Pro, by up to 58.56% on certain Geoperception benchmark tasks and 10.65% on average across all tasks.

Figures (22)

• Figure 1: Four examples from our Geoperception dataset. The questions are sourced from the Geometry-3K corpus geometry3k, which compiles problems from two widely-used high school textbooks. We perform filtering, validation, and generate question-and-answer text for each image.
• Figure 2: Three geometry logical shapes, of increasing complexity, used in our empirical study. Our geometry image generation engine is able to produce infinite visual instances for each of these logical shapes. All letters are randomly sampled from the alphabet and reassigned to each of the points before drawing.
• Figure 3: LLM size experiments. Training loss and testing accuracy curve comparing three choices of LLM size with a fixed visual encoder and multimodal connector. Training losses are window-smoothed using a window size of 10 for better visibility.
• Figure 4: Vision encoder experiments. Training loss and testing accuracy (on a 1500 instances holdout test set) curve comparing eight visual encoders, with a fixed multimodal encoder and LLM. For a fair comparison, all visual encoder transcribe an image into 256 visual tokens. Training losses are window-smoothed using a window size of 10 for better visibility.
• Figure 5: Tuning/freezing vision encoder experiments. Testing accuracy (on a 1500 instances holdout test set) curve comparing freezing versus tuning the visual encoder during training.